{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What Happens When You Predict on an Unseen Categorical Level\n",
    "i.e. a rare event appears in your test set, which wasn't present in your training set.\n",
    "___\n",
    "\n",
    "> *This notebook walks through the steps of importing, cleaning, training, and testing on a data set where the test set contains a categorical level that was not present in the training set. You need to run steps 2. - 6. (to load all the variables) before you can jump between sections and run individual cells*\n",
    "\n",
    "> Up to date for release H2O cluster version 3.8.2.1 and compatible with Python 3.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "0. [Jump to The Short Answer](#question)\n",
    "1. [Import, Visualize & Clean Data](#start)\n",
    "2. [Split into Train and Test Sets](#split) \n",
    "3. [Build Models for GLM, GBM, DRF, Deep Learning & Naive Bayes](#build)\n",
    "4. [Model Performance for GLM, GBM, DRF, Deep Learning & Naive Bayes](#output)\n",
    "5. [How Each Algorithm Handles New Categorical Levels During Prediction](#answer)\n",
    "6. [Bonus Section! How Each Algorithm Handles Missing Values During Training & Testing?](#missing)\n",
    "6. [Additional Resources](#resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"start\"></a>\n",
    "## Import Packages, Initialize an H2O Cluster & Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import h2o, pandas, pprint, operator, numpy as np, matplotlib.pyplot as plt\n",
    "from h2o.estimators.glm import H2OGeneralizedLinearEstimator\n",
    "from h2o.estimators.gbm import H2OGradientBoostingEstimator\n",
    "from h2o.estimators.random_forest import H2ORandomForestEstimator\n",
    "from h2o.estimators.deeplearning import H2ODeepLearningEstimator\n",
    "from h2o.estimators.naive_bayes import H2ONaiveBayesEstimator\n",
    "from tabulate import tabulate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set 'interactive = True' for interactive plots, 'interactive = False' if not:\n",
    "interactive = True\n",
    "if not interactive: matplotlib.use('Agg', warn=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O cluster uptime: </td>\n",
       "<td>8 minutes 15 seconds 464 milliseconds </td></tr>\n",
       "<tr><td>H2O cluster version: </td>\n",
       "<td>3.9.1.3460</td></tr>\n",
       "<tr><td>H2O cluster name: </td>\n",
       "<td>H2O_started_from_python_laurend_kxh246</td></tr>\n",
       "<tr><td>H2O cluster total nodes: </td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O cluster total free memory: </td>\n",
       "<td>3.3 GB</td></tr>\n",
       "<tr><td>H2O cluster total cores: </td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster allowed cores: </td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster healthy: </td>\n",
       "<td>True</td></tr>\n",
       "<tr><td>H2O Connection ip: </td>\n",
       "<td>127.0.0.1</td></tr>\n",
       "<tr><td>H2O Connection port: </td>\n",
       "<td>54321</td></tr>\n",
       "<tr><td>H2O Connection proxy: </td>\n",
       "<td>None</td></tr>\n",
       "<tr><td>Python Version: </td>\n",
       "<td>3.5.1</td></tr></table></div>"
      ],
      "text/plain": [
       "------------------------------  --------------------------------------\n",
       "H2O cluster uptime:             8 minutes 15 seconds 464 milliseconds\n",
       "H2O cluster version:            3.9.1.3460\n",
       "H2O cluster name:               H2O_started_from_python_laurend_kxh246\n",
       "H2O cluster total nodes:        1\n",
       "H2O cluster total free memory:  3.3 GB\n",
       "H2O cluster total cores:        8\n",
       "H2O cluster allowed cores:      8\n",
       "H2O cluster healthy:            True\n",
       "H2O Connection ip:              127.0.0.1\n",
       "H2O Connection port:            54321\n",
       "H2O Connection proxy:\n",
       "Python Version:                 3.5.1\n",
       "------------------------------  --------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Connect to a cluster \n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Import and Parse airlines data\n",
      "\n",
      "Parse Progress: [##################################################] 100%\n"
     ]
    }
   ],
   "source": [
    "# 1 - Load data - One row per flight.  \n",
    "# Columns include origin, destination, departure & arrival times, carrier information, and whether flight was delayed.\n",
    "print(\"Import and Parse airlines data\")\n",
    "# air_path = 'allyears2k_headers.zip'\n",
    "air_path = \"http://h2o-public-test-data.s3.amazonaws.com/smalldata/airlines/allyears2k_headers.zip\"\n",
    "data = h2o.import_file(path = air_path)\n",
    "# data.describe() # uncomment to see summary of loaded data file\n",
    "# data.head()     # uncomment to see top of the loaded data file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore Data with Visualizations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "glm Model Build Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEPCAYAAABsj5JaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXl4VPW5+D9vlkmGhAQoAWRLFFSwLgQF7XULXkSvS/Gn\nrcq1rpGrtaiXamVprdwqdd8t4oKCVhZba21vrUGvRKu3OlTDokSLV4MCQgYXBExIQt7fH98zySSZ\nrMxMZsL7eZ7zzMmZs7xzGM477y6qimEYhmF0lpTuFsAwDMNITkyBGIZhGF3CFIhhGIbRJUyBGIZh\nGF3CFIhhGIbRJUyBGIZhGF0iLgpERFJE5F0R+ZP3d18RWS4iH4pIiYjkhu07S0TWi0i5iEyKh3yG\nYRhG54mXBXItsC7s75nAK6p6MPAqMAtARA4BzgVGA/8GzBMRiZOMhmEYRieIuQIRkaHAacDjYZsn\nA4u89UXAWd7694GlqlqnqhXAemB8rGU0DMMwOk88LJB7gZ8B4SXvA1V1K4CqbgEGeNuHAJ+F7bfJ\n22YYhmEkGDFVICJyOrBVVVcBbbmirJ+KYRhGkpEW4/MfC3xfRE4D/EBvEXka2CIiA1V1q4gMAiq9\n/TcBw8KOH+pta4KImMIxDMPoAqoatbhyTC0QVZ2tqsNV9QDgfOBVVb0Q+DNwibfbxcAL3vqfgPNF\nxCci+wMjgUAr507a5aabbup2GUz+7pdjX5Q/mWXvCfJHm1hbIK1xG/CsiFwGbMBlXqGq60TkWVzG\nVi1wlcbiUxuGYRh7TdwUiKq+BrzmrX8JTGxlv1uBW+Mll2EYhtE1rBK9GygqKupuEfYKk797SWb5\nk1l2SH75o40ko4dIRMyzZRiG0UlEBE2WILphGIbRczEFYhiGYXQJUyCGYRhGlzAFYhiGYXQJUyCG\nYRhGlzAFYhiGYXQJUyCGYRhGlzAFYhiGYXQJUyCGYRhGlzAFYhiGYXQJUyCGYRhGlzAFYhiGYXQJ\nUyCGYRhGlzAFYhiGYXQJUyCGYRhGlzAFYhiGYXQJUyCGYRhGl4ipAhGRDBF5W0TKROR9Efm1t/0m\nEdkoIu96y6lhx8wSkfUiUi4ik2Ipn2EYhtF1Yj7SVkR6qeq3IpIKvAlcB0wEdqjqPc32HQ0sBsYB\nQ4FXgAObz6+1kbaGYRidJ+lG2qrqt95qhne9r7y/I32IycBSVa1T1QpgPTA+1jIahmEYnSfmCkRE\nUkSkDNgClKrqOu+taSKySkQeF5Fcb9sQ4LOwwzd52wzDMIwEIx4WSL2qFuJcUieIyInAPOAAVR2D\nUyx3x1oOwzAMI7qkxetCqvqNiPwFOEpVXwt76zHgz976JmBY2HtDvW0tmDNnTsN6UVERRUVF0RTX\nMIx9gGAwSEVFBQUFBeTl5cXsmO6itLSU0tLS2F1AVWO2AP2BXG/dD7wO/CswKGyf6cBib/0QoAzw\nAfsDH+EF+pudVw3DMNqjsrJSA4GAVlZWtnhv8eKl6vf309zcser399PFi5e2e76uHJNIeM/OqD3j\nY5qFJSKHAYtwAfMU4GlVvUtEngLGAPVABXCFqm71jpkFFAO1wLWqujzCeTWWchuGkfwsWbKM4uKr\n8PkKqKmpYMGCeUyZch7grIj8/FFUVa0ADgfW4PdPYMOGD1q1KrpyTKIR7SysmLqwVHUtMDbC9ova\nOOZW4NZYymUYRs8mGAxSXHwVVVUrqKpyD/vi4glMnHgSeXl5VFRU4PMVeO8BHE56ej4VFRWtKoOu\nHNPTsUp0wzB6HKGHvbMUIPxhD1BQ4KwSWOO9v4ba2g0UFBS0es6uHNPTMQViGEaPo72HfV5eHgsW\nzMPvn0BOzlj8/gksWDCvTUuiK8f0dGJeiR4LLAZiGEZ7hGIg6en51NZuaBIDCdHTs7CaE+0YiCkQ\nwzB6LMn8sI8FpkAwBWIYhtEVkq4XlmEYRncRDAZZuXIlwWCwu0XpkZgCMQyjR7JkyTLy80dx8slX\nkp8/iiVLlnW3SD0Oc2EZhtHj6AlFf7HAXFiGYRjt0F4diBEdTIEYhtHjsKK/+GAKxDCMHocV/cUH\ni4EYhtFjsTqQplgdCKZADMMwuoIF0Q3D2GewOo7ExhSIYRgJSTLUcdTV1fHAAw+wa9eu7halWzAX\nlmEYCUcy1HG89957XHLJJfTr14+nn36agQMHdrdI7WIuLMMwkpLOuKMSuY6jrq6OuXPnMmHCBK68\n8kpKSkqSQnnEAlMghmHEnM66o+JVx9HZGMvatWs55phjeP3113nnnXe4/PLLEYnaD/rkI5oD1uO1\nOLENw0gGKisr1e/vp7BaQRVWq9/fTysrK9s8bvHiper399OcnEL1+/vp4sVLu3TtQCAQ8Vqh8+fm\njm33/DU1NXrzzTdr//799d5779W33367XfkTEe/ZGb1ncTRP1uLkkAG8DZQB7wO/9rb3BZYDHwIl\nQG7YMbOA9UA5MKmV80b5thqGESsCgYDm5o71lIdbcnIKNRAItHtsWwqgPUIKIivriBYKojNKbc2a\nNTp27Fg95ZRT9MEHH+qw0klEkkqBOHnp5b2mAm8BxwK3Azd422cAt3nrh3jKJg0oAD7CC/Q3O2eU\nb6thGLGiqxbI3l7T58ttck2fL7fhmi2VWqVmZR2kJSUlDecItzoef/xx3bp1a9w/R7SJtgKJeQxE\nVb/1VjNwMZevgMnAIm/7IuAsb/37wFJVrVPVCpwlMj7WMhqGETu6o61IWVkZNTV5hAfha2r6U1ZW\nBjSPsSwDDmbXrnrOOmsKS5YsY82aNRxzzDG88cYbvPvuuxQXF7Nhw4aEDex3F2mxvoCIpADvACOA\n+aq6TkQGqupWAFXdIiIDvN2HAH8PO3yTt80wjCRmypTzmDjxpDi3FdmMUxCHe6+fN7wTUmqXXXYi\n1dU1uMfO4VRVvcNFFx1Pbm4v7rjjDi699NKGIHlTpePOua83aIy5AlHVeqBQRHKAEhEpApoXcXS6\nqGPOnDkN60VFRRQVFXVdSMMwYk5eXl7cajgKCwtJT0+htrYI5w2vID09hcLCwoZ9pkw5j+98py9n\nn30Du3aFlMxUVNNYuHAhZ5xxRgv5FyyYR3HxBNLT86mt3ZDwDRpLS0spLS2N2fnjWkgoIjcCVUAx\nUKSqW0VkELBCVUeLyEycj+52b/+XgJtU9e1m59F4ym0YRvKxZMkyLrvsSlJTB7BnTyVPPDGfKVPO\na7JPMBhk+PCDqa4+H/gdcDWZmffx6acftqoYkrlBY1I1UxSR/kCtqm4XET8u4+q/gEnAl6p6u4jM\nAPqq6kwROQR4Bjga57p6GTiwubYwBWIYRkdo72G/evVqJk8+i88+20SvXgexZ8/nLFgwr4Wi6Skk\nmwI5DBckF1wA/WlVvUtE+gHPAsOADcC5qvq1d8wsnIVSC1yrqssjnNcUiGEYXaa2tpZbb72Vhx56\niNtvv53TTz+dDRs2dMiqMAsk7HzJ+CA2BWIYyUeiPHhXrVrFpZdeyuDBg3nkkUcYOnRoh49dsmQZ\nxcVX4fO5gHqyWSumQDAFYhjJRiI8eGtqarj11lv5zW9+wx133MHFF1/cqTYkydDgsT2smaJhGElF\nMBikuPgqqqpWsH37O1RVraC4+Kq4zvhYtWoV48ePJxAIUFZWxiWXXNLpHlaJ3OCxuzAFYhhGTOnO\nB29NTQ1z5sxh0qRJTJ8+nf/+7/9myJCulZbFq8FjMhHzOhDDMPZtuqsAb9WqVVxyySUMHTqUsrKy\nLiuOEMlYBxJrLAZiGEbMufrqa3nooceAocBGpk2byoMP3h+Ta9XU1DB37lwefvhh7rzzTi666KKo\ntlxPlGSArmBBdEyBGEYy0Rh8XgBsB3Lx+4tjEnwOxTeGDRvGI4880q7V0RVlYAqkEYuBGIYRU1ys\now9wGXAXcBmqOVGNgdTU1HDTTTdxyimncN111/HnP/+5XeXRlZnryTCnPZ6YBWIYRkwpLy/nkEOO\nxE1zCPWcOoZ1695h9OjRe33+kNUxfPhwHnnkEQYPHtzuMV1JyW085h7gY+AA/P6fWhqvYRj7LuXl\n5SxatIjy8vKYnH/t2rXAYMKzsGCwt73r1NTU8Mtf/pJTTjmF66+/nj/96U8dUh7QtcywiooKdu+u\nB34MLAF+THV1naXxGoaxb3L11f/JIYccySWX/JpDDjmSq6++NurX2Lp1K66VemP6K3zube8a7777\nLkcddRSrVq1i9erVXHjhhZ0KlHclJXfr1q3U1+/GWVL/BN5CtXavPkeyYwrEMPZRysvLeeihR3EP\nxA+Bt3jooceibolMnDgRqAOKgLHea523vXPs3r2bG2+8kVNPPZUbbriBF154gf3226/T58nLy+Pe\ne28jI+NEevcu7NCQq5UrV+KyyMItqSHe9n0TUyCGsY8SCARw/UzDH4hDve3RY/To0UybdiVuksM2\noIpp067sdPzjnXfe4aijjmL16tWsXr2aH/3oRy2sjmAwyMqVK9utcl+yZBnTp8/E5xtGTc3H3Hvv\nbe22Vhk3bhywkaaW1CZv+z5KNOfjxmvBZqIbxl6zbt06BX+TGd/g13Xr1sXsegsXLuz0+aurq/Xn\nP/+5DhgwQH/7299qfX19xP0WL16qfn8/zc0dq35/P128eGnE/bo6oz0QCGhKSq53z0Yq+FUkRwOB\nQKc+T3dClGeid7sy6JLQpkAMIypMm3aN90A8UMGv06Zd090iNeEf//iHHnroofr9739fN2/e3Op+\nnVEKgUBAc3PHevu5JSensF1F0HiNhQq/VFjYIcWTSJgCMQViGFGlq5ZBe1RWVmogEOjSA7ajVkeI\nziiFrlogqo1WTk5OYZtWTqJiCsQUiGEkPB11J0Vi5cqVeuihh+rkyZP1888/79AxnVUKe6MI9kYx\ndjemQEyBGEbUiaYV0tVf+NXV1Tp79mwdMGCAPvPMM+1aHc3prFJIZkXQVaKtQKwbr2HsQ0Tq43T1\n1f/ppfMOAz7b60aHoSK9qqqWRXqtpcn+4x//4JJLLmHkyJGsXr2aQYMGdfq6U6acx8SJJ3W4T1Ve\nXl7SVJAnLNHURvFaMAvEMDpNJLdSy0ysFQoZ+sYbb3T5Op2xQKqrq3XWrFk6YMAAXbx4caetDqNz\nEGULJKZ1ICIyVEReFZH3RWStiFztbb9JRDaKyLvecmrYMbNEZL2IlIvIpFjKZxj7Co1TAZ9j+/b5\nVFU9R3HxVbzyyivAEFwNyDLgHGA4J510epcbBYbmZvj9E8jJGdtqkd7KlSsZO3Ys5eXlrF69milT\npkS17boRB6KpjZovwCBgjLeejSt3HQXcBPw0wv6jgTLcoKsC4CO8ho/N9ouuWjaMJKc9f34gEFC/\n/wCFfgpjFfppZmaB3nPPPZ4FssJ7r/OZSZ2Vqbq6WmfOnGlWRzdAMlkgqrpFVVd56zuBctzPHYBI\nPzUmA0tVtU5VK4D1wPhYymgYyc6SJcsYPvwgJky4kOHDD4poOWRnZ1NV9TmwAngHWEF19Vaqqqpw\nrdYnA/2J5tjZvLw8xo0b18TyCAQCjB07lg8//JA1a9aY1ZHkxK2ViYgUAGOAt71N00RklYg8LiK5\n3rYhwGdhh22iUeEYhtGMYDDIxRdPpbpa2LXLR3X1Hi66qLhFK4+dO3fi948kXEH4/SMYOXIk8DWw\nCKgkVvO+q6urmTVrFmeeeSY33ngjzz33HAMHDozKuY3uIy5ZWCKSDfweuFZVd4rIPOBXqqoicgtw\nN3B5Z845Z86chvWioiKKioqiJ7BhJAllZWXU1u4BfgHcCgykrm4jt9wyl/vvv69hP6cINhE+lxw2\nM2HCBKZNm8pDD/07zhI5Bp+vgNTUrVGb9x0IBLj00ksZNWoUa9asMcURR0pLSyktLY3dBaLpD4u0\n4JTUSzjlEen9fGCNtz4TmBH23kvA0RGO2XtnoGH0AEpKShT2V+it0NeLb/RVSNeZM2c3iT+01bYk\nVAfyxhtvRK02oqqqSmfMmKEDBw7UpUuXWqwjASCZYiAeTwDrVLUhsVxEwpO8zwbe89b/BJwvIj4R\n2R8YCUS3Nahh9CAKCwtJTd0K7AFKcfGNmUAat932bMPY1WAwyKOPLgJeBJ4BXuTRRxc1cXV98803\nbNq0KSqzvt9++23Gjh3L+vXrWb16Needd57FOnoi0dRGzRfgWNw3exUuu+pd4FTgKZwNvQr4IzAw\n7JhZuOyrcmBSK+eNsl42jORl5szZCgUKAYV1EbOpli1b5nWQrfT2q1QYoSUlJTpt2rWeZTJCoZem\npvZqs5NtWxZKVVWV3nDDDWZ1JChYKxNTIIYRzvz5j3oK4AiFHIXvtmgqOHfuXIUMhVyFg71XX1ga\nb3hL976amdmnhZJor7/VW2+9paNGjdJzzjlHt27dGs9bYHSQaCsQGyhlGElCpGFJwWCQ6dNn4qYK\nrgJeAD6meTbVgAEDvL9TgCxCCZjr16+n5ZS9AqB/kxTexkLEFWzf/g5VVSsoLr6KYDBIdXU1M2bM\nYPLkyfzqV7/i97//fdj1jJ6MKRDDiBMdnZYXiSVLlpGfP4qTT76yIa4Bru9UWlo+jQqgCOgHHAMc\nSEbGiSxYMI+cnBwglcY4SSmQxoEHHkjLKXsVqFby1VdfNcga6m/VvE7kz3/+M4WFhXz88cesWbOG\nH/7wh53+bEYSE01zJl4L5sIykoy9aW/eVm+pyspKTU/P8SrJA95rL89dlaKPPfaYqqoXAxnRIgby\n2GOPqUhGkxgIpGt6ek4TWVvK8LampWVqXl6ePvvss7G6bUaUwVxYhpFctOX+6Qit/foPuZjq6mqA\n04B/914vxyUvZrBjxw4A+vTpg7M0Dgau9F4/429/exPVVGAE8DkuKTKd2tq/NZEVaOhv1avXwYh8\njyOPHMP7779vVsc+jCkQw4gx7SmA9igoKKCmpoJIVeJlZWWopuBiIOu910XAfsAQNm3aBEBWViju\nUUqjCyuFpUufB/4OrPVef+cd21LWs876Ppdeej5+/xc89tijvPXW360d+j6OKRDDiDFtKYCOEOpu\nm5l5IllZB5OZeWJDlfjXX39N8we+62n1HLCpoUPDu+++S2PX3fD9mm8bAWxuIWswGKSwsJBt27ZR\nXl5OcXFxCzn3JsZjJCemQAwjxnS0vXl7iKQAfu/V4VxTn9M0CP45cC0pKb0YOHAgS5Ys42c/u5Hm\nigG2odo8gL4FyCYj40RycsaSmVnESSd9j+LiYm655RaWLVsWUe7WgvxGD6e9IAlwEPA/wHve34cD\nv4hmIKazCxZEN5KQzo5QDW8v0nYQPdtrX1LovfoVHlC/v5+uW7fOO3aFwlyvTmSIQh+FgXrNNaEi\nwsO9AsTb1efL1XXr1unjjz+uI0aM0HPPPbdNmbs6wtaIP8S7kBB4DddSvSxs23vRFKLTQpsCMXo4\nrjo8U2G4gk99vtEtigMDgYC37zXevkO9V6dEJk36t2ZzQA7wlMVIT5Fk6LJly3T+/EfV5+utmZmj\nNDOzry5c+JRed911OmjQIP3d737XrqyBQEBzc8e2Kp+ROERbgXTEhdVLVZv3o6rba9PHMIyIlJeX\n89BD8wE/Lk7hbzWGEgwGeeSRJ4F0nEc6HagFXmT58lK2bt3qzQF5Dte2PRRs/xshD3ZOTg4pKamk\npir19bXMmjWDjRs3snbtWn7wgx+0K+/exniM5KUjCmSbiIwAFEBEfoBzshqGEQPcmNnwor/XgXrS\n049vEUNx7dxrcE2v+3uvtUANMJSVK1d6c0CycBXm4QHzwXzzzTcUF19FdfVL7Np1OjU1mXzxxQ4e\nfPBB+vfv3yF5oxXjMZKPjswD+QnwKDBKRDYBnwA/iqlUhrEP4+ZlhDKrgsBuYD/mzfs5RxxxRJNu\nuS4LK6Rs9gNextWBrAY2MmnSJO688yFgF1BB03kgn1NVVUVKSn/gQuBIoJzMzElUVFR0SgFMmXIe\nEyeeREVFRVS6+RrJQbsWiKp+rKoTgTxglKoep27crGEYMeCwww7DZUzdAYwCpgJbWLv2vVYezvsB\nrwD7A7fgpkXPZtKkIo499ljPOjgH5946BjgM1/KknnfeeYddu9bjrJNpwOYuu58ijbA1ejbi4ipt\n7CDSB7gI9w1rsFhU9ZqYSta2TNqe3IaRrKxcuZITTvgB1dVBXMyiHPgx8B38/i9ZsGAeU6acB8Dy\n5cs55ZQzcP81h+EmQv8/4M9kZqby6af/JC8vjzfffJPjjivCxVX2w00n3EX//nls27bDO3YjUMfi\nxU83nN/oWYgIqhq1wSwdiYG8iFMea3EO2dBiGEYMKCgoQPVr4EDcw/4qnItqfYs2KK7CPA2naD70\nXp8H+lBf72+odneFhPm4/8ZjgWygL9u27QTeBv7pHZvG8OFD4/NBjaSnIwokU1V/qqpPquqi0BJz\nyQxjHyUvL4/7778Lly31Ms2D3+FtUD766CNatmMfCnxBTc1XZGdnA6G4ykbgBNx/+2eBbyIcO4Tl\ny5fH8NMZPYmOKJBnRGSqiOwnIv1CS8wlM4x9mCuumMr8+ffj8/0Y+IDWUmRHjhxJy3bsG4Gb8ftH\nsnPnTr799lv+8pe/4ILxX+BcYmd5+zc/dhOTJk3qstzWzmTfoiMKZDdwF67TWsh99Y9YCmUYhlMi\nGzd+xM03/7zVFFmfz0d6em9C8z/c64+AicBmtmzZwhFHHMHWrVvJzByOC6TvAJTMzGEcfXRhk2ND\ngff2iKQorJ3JvkdHgugfA+NVdVt8RGofC6IbPZ1gMNgkJbb53+H75eePoqpqAbAc+C0wANjESScd\nS3n5OubNm8exxx7r7fccriZkF37/OWzY8AH//Oc/Wb58OZMmTeqQ8liyZBnFxVfh87kCwgUL5jFx\n4kne+VcQShP2+yewYcMHlpWVQEQ7iN6RtiHLcdXoXWk5MhR4FXgfF727xtve1zvvh0AJkBt2zCyc\n87ccmNTKefemmt8wEprFi5dqZmYfzco6WDMz+7Q7fOrmm+d6g6AKvR5XU1XEp6eeeqpu27atyXn9\n/n6ak1PY6aFWIVrre1VSUmLtTJIAotzKpCMWyPPAd4EVOHdWSPG0m8YrIoOAQaq6SkSyce6vycCl\nwBeqeoeIzAD6qupMETkEeAYY5ymfV4ADtZmQZoEYPZVgMMiQISOorU3D1XV8Qnp6LZs2fdzqL/nG\nY+7DZVL9kbS0XWze3LIYMJIlE75t27ZtBAIBxo8fz+jRo1tca+XKlZx88pVs396YiJmTM5bf/e42\nzjprilkgCU53WCAXR1q6oq2AP+Kcsx8AA71tg4APvPWZwIyw/f8KHB3hPNFRx4aRYJSUlHjWROMv\nfOilM2fObvWYyspKTU31K6R4jRRzNT09u0PdcENWSVbWEZqS4veaLR6k4Ndp066JeK3WOu9Gw8Ix\nYgvx7sYbtQu5XMQKXAL6V83e+9J7fRD497DtjwNnRzhX9O6oYSQQToGMbOIKghHq80VWCDt37tRz\nzz1XRdIVntLQvPOOuI8a56k/77V69zdTXH5dt25di+PaUhSdbVlvxJdoK5BWe2GJyLOqeq6IrMVr\npNjMcjk8wmGtnSsb+D1wraruFJHm5+u0P2rOnDkN60VFRQ2T1wwjmSksLCQ9vZLa2vCeVdvw+Qpa\n9Kd67bXXuOyyyxg7diwZGVlUVx/RcExH2pHcc8991Nam4map9wcG07yeJBAItHBltdX3Ki8vz1xW\nCURpaSmlpaWxu0BrmgXYz3vNj7R0VEPhymRfwimP0LZymrqwyjWyC+slzIVl7GPMn/+oZw0c2jDk\nKXxA086dO/Xqq6/WwYMH6wsvvKCqnQ+QV1ZWalpaVpjVUem5v9q3QIzkhXi5sIDlUbkAPAXc02zb\n7SFFAcwAbvPWDwHKAB8ugvgRXqpxs+Oje1cNI8GYP/9RzcjI0ezsQ5sohNLSUj3ggAP0wgsv1C++\n+KLJMZ1xHwUCAc3IGO7FO0KusqVe/GWkhgZSGT2LaCuQVrOwRKRMVQv3xroRkWNxwwxCbjAFZgMB\nXC+FYcAG4Fx1zX8QkVlAMW6owbWq2qKvgmVhGfsC4dlRfr+fWbNm8fzzz/Pwww9z5pln7vW5hw0b\nye7dtbjMrZC77HhcR99h+P3FlkXVw4h2FlZbCuRj4PrWDlTVP0RLiM5iCsTYlygtLaW4uJjjjjuO\n++67j759+0blvEuWLOOCCy5GNQUYgmsh/wTgOvHm5IzllVceYdy4cVG5ntH9RFuBtDVQKhc4Azdc\noDkKdJsCMYx9gZ07dzJz5kz++Mc/Mn/+fM4444yonn/ixJPIyPBTXT0bZ/DfDoQC5jaW1mifthTI\nBlW9LG6SGIbRQGlpKZdddhnHH388a9eujZrVEU5FRQUZGQdQXf0zb8sI4HtkZY2kvn6jjaU12qUt\nBRK9akXDMDpEe1ZHaz2xukJBgetl1TjmdjSZmT7+8Ic7KSwsNOVhtEtb3XgvFJFUEVkRN2kMo4fT\nVrvzFStWcPjhh7Nr1y7Wrl3bQnlEu9ttXl6eN+62sdPvE0/MZ9KkSaY8jA7RkV5Y/4OrBt8eH5Ha\nx4LoRjISqYvtlCnnsXPnTmbMmMELL7zAI488wumnn97i2Mauu9HvNRVNq8ZIbOIZRA+xE1grIi8D\nu0IbtRtnohtGshEMBikuvoqqqhVUVTkFUFw8gYyMdK6//npOPPFE3nvvPfr06RPx+IqKCny+Au9Y\nCJ9MuLcPfaseN7pKRxTIH7CMK8PYK1oqgAOoq0vhxz/+MU8++SSnnXZam8e3jFdYlpTR/bSrQNTm\nnxv7MNFw7wSDQb766qswBbANuJD6+m94+OElHaqzCMUriosnkJ6eT23tBsuSMrqdtgoJ22qmqKp6\nRMylawWLgRjxoLWYRVfPsWvXh9TX16KqpKSkkZrqw+8f0alzW7zC2BviWYm+n6p+LiL54Ztx7Udm\nqWrbNncMMQVixJqOBK3be5g3PUcQuIjU1G089th8fvKT6234khF3oq1AWk3jVdXPvdcNQD9gGlAK\n/Ap4MVoCGEYiEopZhLc3DwWtoWMptRUVFaSnDwMeBi4BHicr67ukpKS0eW7DSBZaVSAicpCI3CQi\n5cB9uKY92ASbAAAgAElEQVSHoqoTVPWhuEloGN1A06A1hAetwzOqtm9/h6qqFRQXX9WituPTTz9l\nx461wBbgPWAItbUbGD9+fKvnNoxkoq1Cwg+AI4FJqnqipzT2xEcsw4gtbRX0QeQiu1DQuj3rZMeO\nHVx55ZVMnz6dG264Ab//dXJyJjScY/To0a2e2zCSibZiIGcB5wNHAyW49usLVHX/+IkXGYuBGHtD\nZ4LjkeIcbcVHVq9ezeWXX87EiRO5++67yc3NbTVWYgFxI97ELYgedsEsYDIwBTgJNyDq+UhzOuKF\nKRCjq0SrojukhEIptQ8+eBeBwFv89a9/5bHHHuOUU07ZazlNuRjRJu6V6Kq6C1gMLBaRvsAPcVME\nu02BGEZXiUZFdzAYZOTIA3j55T/x0UcfUVdXx69+NYdJkyaxdu1acnNz90rGaKQPG0Y8aNcCSUTM\nAjG6yt5aIKGHOwyhqmo9qanp1Nfv4oYbZnDbbb/udvkMoy3ilsZrGD2RvLw8iot/BBwDHAQcQ3Hx\njzr0cG7az+ouoC979tSg+gIPPPBIqwH5ztBegN4wEglTIEbSEZ5B1V42VfPjli9fzuOPP4UrZXoG\neJEFC37boeMb6zoeAqYCC4FDgIFRe8i3lT5sGIlGTBWIiCwQka0isiZs200islFE3vWWU8PemyUi\n60WkXEQmxVI2IzkJL+AbOvRAhgw5oEPzMULHnX32DVRX1wBbgXFAUYcf/p988olX1/ElsBYYhCuP\n2hW1h3xb6cOGkXCoaswW4DhgDLAmbNtNwE8j7DsaKMMF9guAj/BiNBH2VWPfo7KyUv3+fgqrFdR7\n7atQqbBa/f5+WllZ2cHjchXWKaxWny834nEhvv76a7388st1+PDhOnPmLPX7+6nff6iCX32+/TQz\ns48uXrw06p81EAi0KZdhdBbv2Rm1Z3xMLRBVfQP4KsJbkYI4k4GlqlqnqhXAemB8DMUzkoxI8QH3\nW6OCtmIFkY/LAwqBf0G19frYkpISDjvsMFJSUli7di233vprNmz4gNmzp5CZmUF6+gBEUvjmm2+6\n5FZrjby8PMaNG2eWh5HQdGQeSCyYJiIXAv8ArlM37XAI8PewfTZ52wwDiDwTwymPAkKxguzsbFau\nXNmkfiI7O5vq6o+bHfcl8BJwDhkZg1uk8W7fvp3rrruOV155hQULFjBmzBg+/PDDBjfVr399N9XV\nr3nnu4Mrr7yW3r1HUV39ESIpne6yaxjJSHcokHnAr1RVReQW4G7g8s6eZM6cOQ3rRUVFFBUVRUs+\nI0FpPhOjuvpj6uvryMg4jj17KikuvpgjjzyuSf0EQHHxVdTVZeEyr4bglMc8oAgYyu7dH5Odnd1w\nnZKSEqZOncppp53GmjVr+Mtf/kp+/qiG886efV1YLUkQuB14ix079gMOBkqpqWmcOjhx4klmSRjd\nQmlpKaWlpbG7QDT9YZEWIJ+wGEhr7wEzgRlh770EHN3KcVHwBhrJSig+MH/+o+r399OsrCM0M7OP\n+ny5TeIcmZl9msU+nlfIUFgRFgvxa0bGEM3IyNF7771Pi4uLNT8/X19++eWGazWPn/j9/TQzs493\nnoUK3/XeCyiM9dbdkpNTqIFAoJvvmGE4iHIMJB4WiBAW8xCRQaq6xfvzbFybUoA/Ac+IyL24n4kj\ngUAc5DOSjNCv+RNPPDWs4G4xLj+jMc6RkjIAVV/YtrOA/YD/B/THeUlPY/fuFcB3mD59OieccAJr\n166ld+/egIufpKUNAXbjrI3DSU0dzPjx+/Hqq6cBQ4GNwB3ApcAn2NhZY18h1mm8i4H/BQ4SkU9F\n5FLgDhFZIyKrgBOB6QCqug7XsHEdLkn/Kk9jGvsIrQWfI21vGRg/GdiMe3gHgcV8++0GqqpCsQ+8\n12+AJ4EN9OqVD/wPcAJQDzzC22+vprq6uuE67767ih07PsLVfYwCLmDnzo949dU3gbeAf3qvc8jO\nPon09Fp8vhMsBdfYN4imOROvBXNh9TgWL16qfn8/zc0dq35/v4a02Na2R3Itpab28lxUvRRGKGQr\nTFbIUshX8CsMUPCrSIampfkV0hWuVPhGQTU7+4gGl1Pk9F+/wsMtXFW9e4/RhQsXamVlpaXgGgkL\nUXZhdbsy6JLQpkB6FK3FGdatW9fq9kAgoHfeebdmZORor16HaEZGropkKOSE7X+798AfEaY8ein0\nUpEszc3NVfC1UBDz5z+qqqqBQEBzc5sqCjhQoUShpVymMIxEJ9oKxFqZGN1Oa/2fAoFAhPqNwRQW\nHsOJJ57Pz372C3bvzuPbb/+P3btrUB0G1AHlhGdHuZrUt7z3bgSqUK3m4osvISUlBZeddQQwAZjD\n9OkzCQaDEduKuHiHj8YsrpHmqjL2WUyBGN1Oa/2fIo1+rar6P3bvXkRV1dc4pfB3oJe3vt77+8e4\npgbDaFQ+w73Xu3HJfwN54IFHqa9fgMvXeBw3hPOGJgWJs2df16StyLRpU/H7zyEn53YyM5Wbb76M\nDRs+sFoPY98kmuZMvBbMhdXjCMU6cnIKI8ZAsrMP1fT0bM3IGKLwgMJBXguTlqmzzmU1wHNbrVb4\ni8JAhVSFg73tqQqHe+do6Y4KpQjn5o7VzMw+evPNcxtcVBbjMJIVouzCsnkgRsLQ2hS+Rx55jGuv\nvQGRIVRXfwSk4tJxv8BZGw8DoarwNbgOOClANq5oULy/bwZu8PY52jv7X3GNFX8M9MPv/4p7772N\n6dNn2kwOo8dh80CMHkuk/k/BYJDp02eye/drVFevwLmr/o6La7wG3A/sxMUxDiQ19V9IS/Ph6jIy\ngDNwMYsROOUBLkaSgis3Os3bXsXMmeexYcMHjB07pkMzOaLR88owkhlTIEZC0zTAXgHsT9Og+hDA\nD+wBKvjJT4pxHXruAZ4C/ggcQNMakR/TqITeAoLceedcbr11Lnl5eR2ayRHeVr69VvKG0WOJpj8s\nXgsWA0lqOhNDaJriW6mufXt42m0/r6VILwW/9u7dW1NS0hVe9OIjK7x9rvNiHyO9pTFm0rv3mBbt\nRlqLybSUqTFuYjERI9HB0niNZCb0y33ChKlNfrm35g565ZVXqaurAb7nLbsIuatc2u1tOOtDgCp2\n7NhNfX02cA5wAc5FtQeYx9ixh+LzbaHRGgFYQ13dpy3ajUyZ4txZr7zySIssKxs7axge0dRG8Vow\nCyQpaG5pVFZWtmh46PPlNmmKGMqACgQCzQoJKxWeUcj0MqwyFG70qs3Tvayq9DCLI9xK6a3p6Vla\nUlKi69at05tvnquZmX01K+vwFtZFRz+XWSBGMoJVopsCSQYitSApKSlp4T6CEZqamtWierx370LN\nyOijqakDPeXRuP/gwUM8hZHiKZPnveN7eVXiLdN6U1MztXfvw9Tv76fTpl2rmZl9NCvr4C5PE2zL\nxWUYiYopEFMg3ULX4xba8At97ty53kM+3DropbC/93fLmgz3fq7CUm3sRZWmrgVJmsIR3jFLvfqP\nhyOcw6+u5Xo/z2rxazSsB6sHMZINUyCmQOJOaw0NWyNyD6kR2qvXCO/B31ehUKGvpqb6w5RKwFMI\n4ccVeq6rUKPEDM/yaN7Dqq9Cb085DNfGHliZCtM0NDcd+igc0OQaNrPD2FeItgKxILrRJsFgkOLi\nq6iqWsH27e9QVbWC4uKr2qx9iNxD6ku+/fbvuGK+Knr12kVmpvKb39xPenoKrq/UZbj26OHHbcDN\nLs/BpefWeq/5NE3n7QdcDKj3fhou7TcFN1FgFK7+YxhuDkjrKbqGYXQMUyBGm3Ql4yg0etbvn0BW\n1hG47KmHgTzgBrKyRjJv3mw+/fSfXHHFVBYtepzMTCUrq4a0tBRSU4/F9ac6ATgdGANswzVDTANK\ncFXo4Yrmc1w/qxQaJzX7cPUe7wArcPUf6ykuvrBJfytrhGgYXSSa5ky8FsyFFTf2JuOosrJSS0pK\nvPGvkVumh+8byrxKS8tSN8NjgAIKxyrsDIuJVHoxj37qgvJ+hb6anj5Afb4c7dXrIM/F9V1t7kab\nOXN2k+tZ/MLYl8BiIKZA4s3eZhzNn/+o95A/3ItBFKvPl63r1q1rsa/L1PIrnObFMJ7wFEVlgxKA\nYi9e8ry6NN6FCn01M7OPrlu3ThcuXKhZWaNbBNMzMvqYwjD2aaKtQOIxE91IcqZMOY+JE0+K2Ogw\nnFAzxOzsbHbu3Nnwuv/++WRlFbBr13HAb4HXqKmp57DDxvH00wuanPt///d/gRpcoeCzQBZwHy6e\n8Tmu8eFi4HXcbI4U4CrS01N44onHGT16NP3796e+/qfADFyx4VBgPffff7+5qgwjmkRTG8VrwSyQ\nDhFPN03ISvH7D1Dwa1qay4Ty+Q7R9PQcdUV+zVN4+2paWpb6/f20d+/DNTXVp3l5eZ4Lq3ka7qGe\n9ZLb4r177rmnxWcMbwOfkZHTwmVmGPsiJJMLC1iA+8m4JmxbX2A58CEuGpob9t4s3FSgcmBSG+eN\n6k3ticyf/6hmZPTR3r1jX+jWGCdZoY29qZo/6LM9F5aGLUd4abn3Kuyn8CPNzOwbVpl+uGZm9vUU\nyjPqigSbp/keqAsXLmxVLotzGEYjyaZAjsOl0IQrkNuBG7z1GcBt3vohuDFyaUABrlWqtHLe6N7V\nHkZjzCE+rTYa6z5Cw50iVZwXtJDJ1Whkefu+ruE1GeEP/2nTrvGO3T/COfwRYymGYbQk2gokpjEQ\nVX1DRPKbbZ4MnOitLwJKgZnA94GlqloHVIjIetxkoLdjKWNPIxgMcu211wMHEZ56m5Y2nIqKipjE\nALKzs6mu/hjX6LAC988aali4H/AyLu02FVfvUYCr99iNi2H8FjfgydVk1NTUsHTpUvx+P5988gmP\nP/4ULu6xHXgA10xxCLCJSZOKGD16dNQ/k2EY7dMdQfQBqroVQFW3iMgAb/sQXNJ+iE3eNqMTuLqN\nfHbv/gz3AHcT9WpqKmJSLLdkyTKKi69CtReuZuM7wO9x3XGPBepxSkSA/rgJgNcAX+O+fl+SlnYy\nvXqNpLZ2A8cffzTHHXcy0MfbZzBO0VyImwXyT1x9Rw2QxooVfyMYDFpw3DC6gUTIwtKuHDRnzpyG\n9aKiIoqKiqIkTnJTUFBAXd0mYpmBFMq2qqmp4dJL/4Pdu2/BVYw/jCv464t7+O/BDWwKjZodh1Mq\nFwG/Av4NqGLhwsc56KCDqKmp8ZTHi7h27OHHngj8ArgUeKNhe23t9ygrK2PSpElR+WyG0ZMoLS2l\ntLQ0dheIpj8s0oLrOREeAykHBnrrg4Byb30mMCNsv5eAo1s5Z5Q8gj2TWGYgNc+2cnUZfm1ssd7b\n25bhvapCUOF8LxMr3Yt59PLiHz5dtmyZqqouXLhQ4aCwWEp4DOVwhQcixFZGaElJSdQ+n2H0ZEjC\nXljiLSH+BFzirV8MvBC2/XwR8YnI/rheFoE4yNfjCA1DevXVJ/jss4+44oqpUTlvY1+s56iq+hpn\nIYTGwlbhYhxveNtewsVB7sFZC+m4eeaPAZ8BU4GVwHC2bt0KwPjx4733QrGU5j2xBuIl9TVs9/m2\nUVhYGJXPZxhGJ4mmNmq+4CKfm3FO7E9x/oe+wCu4NN7lQJ+w/Wfhnj6WxttNtJX62jLbKtwSGBZm\ncYSsjl4KEpaBNdzb9h1vva+Cr0kWVWPG1aBmFo7Ps058Cn71+w+1ORyG0UmIsgUi7pzJhYhoMsqd\n6IQC4j6f66a7YMG8JqNcg8Eg+fmjqKp6DhejWEHTGMUenCH5NnAHzpJQXAD8dzgL4jNvv98CA4DT\nWLfunSaZVOXl5bzyyiv4/X4Apk692jtn6FpHs2zZIiZMmGDBc8PoBCKCqkr7e3aQaGqjeC2YBdIh\nojEEqqSkpMnxrkAxRzMyhnmWwUhtHOg0VRtHy/rCXlvWbkCoV9UgnTt3bqtyBQIB9fsPa2Lt+P2H\n2vwOw+gCJGEMxOgGlixZRn7+KE4++Ury80exZMmyNveP1La9qqovZ599TcPxS5YsY/r0mfh8I4Ad\nTJjwLziL4jmctfE4cAGuXmMlro/V07hMsPDZHQfiakVWANuprKxsVS6Xetx0fgdstvkdhpEIRFMb\nxWvBLJAWhFsbXWnBHukYF6Nwk/wyM/u0eN/1uErzLI00hSHN4iJjFGZEsECadtedPXt2m5/N5o8b\nRnQgmSrRjfjQPHYxe/Z1+HwFVFW1HALVWswgNASquHgCKSlD2bXrI+AJ3BCoPFJTBwB+wi0J1Rzg\nG1z21X40DnkKxSo+xGVN1RFePQ5zvPO6QVD7779/m5+vo92ADcOILxZET3IaA9uNAe3MzBMRSWmy\nze+fwIYNHzR5+IYKAsMfysFgkLKyMiZPPo/q6teanHPPnnpqa/+GUxY/wrUoSceNi/0U+FdcEeBA\nXOv1VO+9T4CduDYmr+FcWwW4VN1vKSn5kxUCGkYciHYQ3WIgSU6k2IXPtz+zZ1/X5tjWSDGSYDDI\nihUr+PDDD7n55hvx+yeQnT0Gn+8EZsyYTm3tbtx42sG4FmZpuFjHapxieBGnULbg4h4ZwA3evum4\nTjXpuPYmu7xXZdiwYTG9R4ZhxAazQJKcSBZIyNoAIrp9Ih2Tnn4ce/bsob4enIWxmdGjR1Bevh7n\netqAe/jX4JTDl7hGAv+LsyQKcBbGJzRtQTIB+ACneBQ4DRdsHwx8SWZmH15//VnGjRsXoztkGEaI\naFsgFgNJcsJjF+np+dTWbmhhbVRUVDTsG/rbxUjScdlQI6mtrQMycdaCe/iXlx8D3Az0A64CanFW\nxinAH4FzcRlVI3CKo5qWGVf5OFdX0Dv+aVwdyK3AAETOsYwqw0hSzALpIUSKZ0QqDJw48STKyso4\n9dTJuFs4AKjEtRkpAN4JO+v+uCD4bpw7aiguUH4armAwD2eJPAyMxjVKbN5AMWR57AF6A7tISYHs\n7NENyi68WNEwjNgRbQvEFEgPJBQIP+usKU3cVD7fCaSkCKmpQ9m1az1OaeyPsx524bKsXgcOA+7C\nxS9OwSmE173zlAKnAkuBs2jppqoAcnDKKBQ8h9/+diF1dXWMHz+e/v37W0aVYXQD5sIy2iRkdaSk\n5FFV1Y9wd1JNTX/gCpxV8THh7ir38N+BS7eFxuaIn+AsjcOBZThX1nBcW7PdwHk0uqk2k5qax549\n3+IslXrgO/zwh0VccMEFTeQ0xWEYyY9ZID2IpsHx/YCDaaokjsG5k/rhrI2Pwo4eAXzlLWk4pbAZ\n1/PqORpndIT3v5rgvXc6bshTBqmp29mzJ9M7fgNpabVs3vyJKQzDSAAsjddolaYpvXm42MT3yMo6\ngoyMULPDvwJP4uIeofYgK3CupzScIliES7l9C6cg+uEmEfenaYC8H3Am8HPgeZwFk4JL6V0FvEZK\nihm5htFTMQXSgygocMFyZ3WsBAaSkZHGrbdezi9/+TOcAjgHuB6XMXU0MAxXAHgCrqo8DbgbVwBY\nTmPg/Bc0VTprcFXl9TjX1mlALZmZ+YQrmczMAxqywAzD6FmYC6uHcfXV/8lDDz2KUwwbSEvzkZV1\nINXV/8fu3TU0Zkg9jFMk/YHrgJOBo4C/4CrFdwFn42Ih6p3vM9xvjv1wleZ1uBqRz4E0MjJcVtbu\n3X+jrQp4wzC6B3NhGa0SDAZZsOBpnJJ4A+hFXd0bbN/+Drt3z8MV7x0GPIpzO1XjrI7HccqjN85C\nudJ79eFqNwLAehoHRH4CZOOC7DNwKb3/TUrKN9x//11tVsAbhtFzMAd1D8K5iobgfv2vxKXohtxJ\nJwOX4xTIB9571TjX00s0pt2G13AcE3Y+vNfBPPbYTNLT0wkGv+CXv/x5kwLGKVPO4+yzz7I0XcPY\nBzAXVg/izTff5Ljj/hWnEL5LYxbWYbgJgTNxvxn+gpssvAn4d5zS2I2bU74q7Iwjce6pvxNeS7Jx\n4/omzRdNWRhGcmCFhJgCicRdd93DzJlz2LNnELAR1xG3EhfkTgW+9fYciOtnVYCrBfHj0nWDuMB5\neJru93BurN1kZOxPSsoWqxw3jCSmxxQSikgFbnRdPVCrquNFpC8upScfl1d6rqpu7y4Zk4XLL/8P\nFiz4LY3upz8C5+NubSZwBC7Fdh3uljd3U5XiGiHO8P4+GNc8sY6ZM/+Tiy76ETt37jQrwzCMJnSb\nBSIiHwNHqupXYdtuB75Q1TtEZAbQV1VnRjjWLBCP8vJyDjmkEGc9rAL+A1fHkYbLoEoDcnE9q9Jx\nbqn3ws4wCufK2h9XWCi4YPtGzj13MsuWLY3TJzEMI9b0pCwsiXD9ybinH97rWXGVKAkJBAI4t9RH\nuDYjC3GWRz1OeaQBfXDKYzAuphFey7EVl121wDvPk8BiYBDnnHN2vD6GYRhJSHcqEAVeFpGVInK5\nt22gqm4FUNUtuFaxRhsEg1/g4hd9gUdw3r+/A2/jlMeLuBTcl3CDnmbgWpAchnNXXY3rpJuBs1Im\nABmkpX3BhAkT4vpZDMNILrozjfdYVf1cRPKA5SLyIU6phNOqn2rOnDkN60VFRRQVFcVCxoQmGAxy\n4423ADfiuuf2xwXBP8cV+w3BFQWCi3EMBH6Fs06+wt3eu3AWx2ZEIDPzJFQ38cQTj1m8wzCSnNLS\nUkpLS2N2/oTIwhKRm3B9vy8HilR1q4gMAlao6ugI+1sMBJgxYyZ33PEQLn5xL65yPI2mLdpLcMpj\nDXA8znMYas0eCqL/OyJLeP/9f1iw3DB6MD0iBiIivUQk21vPAiYBa3ElzZd4u10MvNAd8iUawWCQ\nlStXEgwGAaisrGTGjBncccftuHYiT+IURh0uo+od7zUV+Ddc4PwYXBykeUPEIaSnL+OZZ55g9OjR\njBs3zpSHYRgdortcWAOB50VEPRmeUdXlIvIP4FkRuQyXR3puN8mXMDSfKnj++WezaNEiVFNxKboX\n4iwLxWVUNVUO7hY+gIuFhIoL1xCyQDIytlFWFmD06BaGnmEYRpskhAurs+wrLqym8z0Ow/WvuhWX\nqvsgriVJEU4fh/pahRcCHgP8F256YGhU7TLgMny+fFJTt1phoGHsQ/SYQkKjfULzPaqqBDgRV2E+\nDJdtBU5R5OJqLv24ViUTcC3Y13v7/BLnqQxZHaPJzPTxwgv3UVhYaO4qwzC6jCmQBCY/P58dO97D\nPfj74mIcu3HxjVDL9c9xSuUWXB3IYOBD3PCoVGAPAwbksWPHhLCmh/OZNGlS3D+PYRg9C1MgCcrm\nzZs59dR/o76+Fhfr+A6ut5XgOugOxVkkR+Gsi9E4l9bLwE+APhx2WA6//vVczjjjDGt6aBhG1LEY\nSIKhqjz99NNcd911fPnlN9TX98KNiD0cZ3mcRtNeVt/DBdD34ILoG4EZ+P232yAnwzCa0CPSeI3I\nbN68mTPPPJO7776b//qv/wIG4QoCQ5lVWTjLIzzT6mDcQKhU4AMyMwfg999ug5wMw4g5pkASAFVl\n0aJFjBkzhqOOOorrr7+Ba665gfr6LbgAeah31S6chRHey2oDbljUEGbPvp7XX3+KDRs+sMwqwzBi\njsVAuplNmzZxxRVXsHHjRpYvX86QIUPIzx/Fnj2PAtfg5nFMwPW42oCLhxwPuPnjbrb558Amjjji\nCMaNG9c9H8QwjH0Os0C6mQsuuIBx48YRCAQYM2YMFRUVpKQMw1kVtcDXwHO41N3ncPUeL+IqzKuA\nXwDHkJKi1vzQMIy4YkH0bqa2tpb09PSGv5sWD5YDF+CskFDWlQJDSU3dAtSTnt4f1a958slHzW1l\nGEabWCFhDyNceYSYOvVCHnjgGFxNRxque+6FwA9w8ztO5bXX/oeDDjrIUnMNw+g2TIEkEEuWLOPi\ni6dSW7sH155kC3Adbt7HrcDTwEamTbuCY489FsAUh2EY3Ya5sBKEYDDI8OEHUV0tuHqPUJ3HBOA5\nMjImc+edtzBx4kRrfGgYRpcwF1YPpaKigtTUgbhaj/A6j35kZEy2GIdhGAmHKZAEoaCggD17tuJa\nlTRvt/6WWR2GYSQclsabIOTl5fHEE/NJT6/FtScZic93Ak8+Od+Uh2EYCYnFQBKMYDBIWVkZgLVb\nNwwjqkQ7BmIKxDAMYx/BmikahmEYCUFCKhAROVVEPhCRf4rIjO6WxzAMw2hJwikQEUkBHgJOAb4L\nTBGRUd0rVXQpLS3tbhH2CpO/e0lm+ZNZdkh++aNNwikQYDywXlU3qGotsBSY3M0yRZVk/xKa/N1L\nMsufzLJD8ssfbRJRgQwBPgv7e6O3zTAMw0ggElGBGIZhGElAwqXxisgxwBxVPdX7eyagqnp72D6J\nJbRhGEaS0KPrQEQkFfgQ+FfcqL0AMEVVy7tVMMMwDKMJCdcLS1X3iMg0YDnOxbbAlIdhGEbikXAW\niGEYhpEcJHwQXURuEpGNIvKut5wa9t4sEVkvIuUiMils+1gRWeMVIt7XPZJHJhmKJEWkQkRWi0iZ\niAS8bX1FZLmIfCgiJSKSG7Z/xH+HOMq7QES2isiasG2dlre7vjetyJ8U33sRGSoir4rI+yKyVkSu\n8bYnxf2PIP/V3vZkuf8ZIvK293/1fRH5tbc9PvdfVRN6AW4Cfhph+2igDOeGKwA+otGiehsY562/\nCJzS3Z/DkyXFkzMfSAdWAaO6W64Icn4M9G227XbgBm99BnCbt35Ia/8OcZT3OGAMsGZv5O2u700r\n8ifF9x4YBIzx1rNx8ctRyXL/25A/Ke6/d61e3msq8BZwbLzuf8JbIB6RsgYmA0tVtU5VK4D1wHgR\nGQT0VtWV3n5PAWfFR8x2SZYiSaGldToZWOStL6Lxnn6fCP8O8RAyhKq+AXzVbHOn5O3O700r8kMS\nfO9VdYuqrvLWdwLlwFCS5P63In+o7izh7z+Aqn7rrWbg/t9+RZzuf7IokGkiskpEHg8zxZoXHG7y\ntg3BFR+GSKRCxGQpklTgZRFZKSKXe9sGqupWcP/pgAHe9tb+HbqbAZ2UNxG/N0n1vReRApwl9Rad\n/1lHV+EAAAR2SURBVL4kkvxve5uS4v6LSIqIlAFbgFJVXUec7n9CKBARednzvYWWtd7rmcA84ABV\nHYO7QXd3r7T7BMeq6ljgNOAnInI8TqmEk2zZF8kmb1J970UkG/g9cK33Sz6pvi8R5E+a+6+q9apa\niLP8jheRIuJ0/xMijVdVT+7gro8Bf/bWNwHDwt4b6m1rbXsisAkYHvZ3IsnWgKp+7r0GReSPOJfU\nVhEZqKpbPXO30ts9Ue93Z+VNqM+hqsGwPxP6ey8iabiH79Oq+oK3OWnufyT5k+n+h1DVb0TkReAo\n4nT/E8ICaQvvw4c4G3jPW/8TcL6I+ERkf2AkEPDMte0iMl5EBLgIeIHEYCUwUkTyRcQHnI/7HAmD\niPTyfo0hIlnAJGAtTs5LvN0upvGeRvx3iKvQDqGpz7pT8ibA96aJ/En2vX8CWKeq94dtS6b730L+\nZLn/ItI/5F4TET9wMi5IHp/7H48sgb3MMHgKWIPLWPojzrcXem8WLougHJgUtv1I3ENvPXB/d3+G\nZp/nVFymx3pgZnfLE0G+/b17Xebdw5ne9n7AK57sy4E+7f07xFHmxcBmYDfwKXAp0Lez8nbX96YV\n+ZPie4/L+NkT9p151/uOd/r7kmDyJ8v9P8yTuQxYDVzvbY/L/bdCQsMwDKNLJLwLyzAMw0hMTIEY\nhmEYXcIUiGEYhtElTIEYhmEYXcIUiGEYhtElTIEYhmEYXcIUiLFPIyJ7vHbd73ktsX8a9t6RbbW1\n9gpCp8RHUsNIPKwOxNinEZFvVDXHW+8PLAHeVNU5HTi2CLhOVc+MqZCGkaCYBWIYHqq6DfgPYBqA\niJwoIn8OWy/zrJV3vDYvtwLHeduu9SyS10XkH95yTNixK0Tkd94Qn6dD1xSRcSLyptf19S0RyfK6\nq94hblDQKhGZGv+7YRjtkxDNFA0jUVDVT7wHeF5ok/d6HXCVqv5dRHoB1cBMnAXyfQARyQQmqmqN\niIzEWTPjvOPH4Ib5bAHeFJF/wfVGWwr8UFXf9XqQVQPFwNeqerTXM+1NEVmuqhti/fkNozOYAjGM\nlkQaJPQmcK+IPAP8QVU3uZ5zTfABD4nIGFx/pQPD3guo1+VYRFbhpsF9A2xW1XehYaAR4saMHiYi\nP/SOzfHOZQrESChMgRhGGCJyAFCnrpV9w3ZVvV1E/hs4HWcRRJr9Ph3YoqqHi0gqUBX23u6w9T00\n/t+LpKwEuFpVX96Lj2IYMcdiIMa+TngL9TzgYeDBFjuJHKCq76vqHTjX0yhgB846CJELfO6tX4Sb\nUd0WHwKDRORI7xrZnuIpAa7y5lQgIgd6rboNI6EwC8TY18kUkXdx7qda4ClVvTfCfv8pIhNw1sP7\nwF9x8ZE94saJLgR+A/xBRC4CXgJ2tXJNBVDVWhE5D+f28gPfAhOBx3Eurne92QyVxGm+tmF0Bkvj\nNQzDMLqEubAMwzCMLmEKxDAMw+gSpkAMwzCMLmEKxDAMw+gSpkAMwzCMLmEKxDAMw+gSpkAMwzCM\nLmEKxDAMw+gS/x8S4/4qcCR5aQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10f64cc88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 2 - Data exploration and munging. \n",
    "# Generate scatter plots of various columns and plot fitted GLM model.\n",
    "\n",
    "# Function to fit a GLM model and plot the fitted (x,y) values\n",
    "def scatter_plot(data, x, y, max_points = 1000, fit = True):\n",
    "    if(fit):\n",
    "        lr = H2OGeneralizedLinearEstimator(family = \"gaussian\")\n",
    "        lr.train(x=x, y=y, training_frame=data)\n",
    "        coeff = lr.coef()\n",
    "    df = data[[x,y]]\n",
    "    runif = df[y].runif()\n",
    "    df_subset = df[runif < float(max_points)/data.nrow]\n",
    "    df_py = h2o.as_list(df_subset)\n",
    "    \n",
    "    if(fit): h2o.remove(lr._id)\n",
    "\n",
    "    # If x variable is string, generate box-and-whisker plot\n",
    "    if(df_py[x].dtype == \"object\"):\n",
    "        if interactive: df_py.boxplot(column = y, by = x)\n",
    "    # Otherwise, generate a scatter plot\n",
    "    else:\n",
    "        if interactive: df_py.plot(x = x, y = y, kind = \"scatter\")\n",
    "    \n",
    "    if(fit):\n",
    "        x_min = min(df_py[x])\n",
    "        x_max = max(df_py[x])\n",
    "        y_min = coeff[\"Intercept\"] + coeff[x]*x_min\n",
    "        y_max = coeff[\"Intercept\"] + coeff[x]*x_max\n",
    "        plt.plot([x_min, x_max], [y_min, y_max], \"k-\")\n",
    "    if interactive: plt.show()\n",
    "        \n",
    "# generate matplotlib plots inside of ipython notebook        \n",
    "%matplotlib inline  \n",
    "scatter_plot(data, \"Distance\", \"AirTime\", fit = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">  Month</th><th style=\"text-align: right;\">  sum_Cancelled</th><th style=\"text-align: right;\">  nrow_Year</th></tr>\n",
       "<tr><td style=\"text-align: right;\">      1</td><td style=\"text-align: right;\">           1067</td><td style=\"text-align: right;\">      41979</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     10</td><td style=\"text-align: right;\">             19</td><td style=\"text-align: right;\">       1999</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows:2 Cols:3\n",
      "\n",
      "Chunk compression summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>chunk_type</b></td>\n",
       "<td><b>chunk_name</b></td>\n",
       "<td><b>count</b></td>\n",
       "<td><b>count_percentage</b></td>\n",
       "<td><b>size</b></td>\n",
       "<td><b>size_percentage</b></td></tr>\n",
       "<tr><td>C1N</td>\n",
       "<td>1-Byte Integers (w/o NAs)</td>\n",
       "<td>1</td>\n",
       "<td>33.333336</td>\n",
       "<td>     70  B</td>\n",
       "<td>30.434782</td></tr>\n",
       "<tr><td>C2</td>\n",
       "<td>2-Byte Integers</td>\n",
       "<td>1</td>\n",
       "<td>33.333336</td>\n",
       "<td>     72  B</td>\n",
       "<td>31.304348</td></tr>\n",
       "<tr><td>C2S</td>\n",
       "<td>2-Byte Fractions</td>\n",
       "<td>1</td>\n",
       "<td>33.333336</td>\n",
       "<td>     88  B</td>\n",
       "<td>38.260868</td></tr></table></div>"
      ],
      "text/plain": [
       "chunk_type    chunk_name                 count    count_percentage    size    size_percentage\n",
       "------------  -------------------------  -------  ------------------  ------  -----------------\n",
       "C1N           1-Byte Integers (w/o NAs)  1        33.3333             70  B   30.4348\n",
       "C2            2-Byte Integers            1        33.3333             72  B   31.3043\n",
       "C2S           2-Byte Fractions           1        33.3333             88  B   38.2609"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Frame distribution summary: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>size</b></td>\n",
       "<td><b>number_of_rows</b></td>\n",
       "<td><b>number_of_chunks_per_column</b></td>\n",
       "<td><b>number_of_chunks</b></td></tr>\n",
       "<tr><td>127.0.0.1:54321</td>\n",
       "<td>    230  B</td>\n",
       "<td>2.0</td>\n",
       "<td>1.0</td>\n",
       "<td>3.0</td></tr>\n",
       "<tr><td>mean</td>\n",
       "<td>    230  B</td>\n",
       "<td>2.0</td>\n",
       "<td>1.0</td>\n",
       "<td>3.0</td></tr>\n",
       "<tr><td>min</td>\n",
       "<td>    230  B</td>\n",
       "<td>2.0</td>\n",
       "<td>1.0</td>\n",
       "<td>3.0</td></tr>\n",
       "<tr><td>max</td>\n",
       "<td>    230  B</td>\n",
       "<td>2.0</td>\n",
       "<td>1.0</td>\n",
       "<td>3.0</td></tr>\n",
       "<tr><td>stddev</td>\n",
       "<td>      0  B</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>total</td>\n",
       "<td>    230  B</td>\n",
       "<td>2.0</td>\n",
       "<td>1.0</td>\n",
       "<td>3.0</td></tr></table></div>"
      ],
      "text/plain": [
       "                 size    number_of_rows    number_of_chunks_per_column    number_of_chunks\n",
       "---------------  ------  ----------------  -----------------------------  ------------------\n",
       "127.0.0.1:54321  230  B  2                 1                              3\n",
       "mean             230  B  2                 1                              3\n",
       "min              230  B  2                 1                              3\n",
       "max              230  B  2                 1                              3\n",
       "stddev           0  B    0                 0                              0\n",
       "total            230  B  2                 1                              3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>       </th><th>Month            </th><th>sum_Cancelled    </th><th>nrow_Year        </th></tr>\n",
       "<tr><td>type   </td><td>int              </td><td>int              </td><td>int              </td></tr>\n",
       "<tr><td>mins   </td><td>1.0              </td><td>19.0             </td><td>1999.0           </td></tr>\n",
       "<tr><td>mean   </td><td>5.5              </td><td>543.0            </td><td>21989.0          </td></tr>\n",
       "<tr><td>maxs   </td><td>10.0             </td><td>1067.0           </td><td>41979.0          </td></tr>\n",
       "<tr><td>sigma  </td><td>6.363961030678928</td><td>741.0479066835018</td><td>28270.12911183817</td></tr>\n",
       "<tr><td>zeros  </td><td>0                </td><td>0                </td><td>0                </td></tr>\n",
       "<tr><td>missing</td><td>0                </td><td>0                </td><td>0                </td></tr>\n",
       "<tr><td>0      </td><td>1.0              </td><td>1067.0           </td><td>41979.0          </td></tr>\n",
       "<tr><td>1      </td><td>10.0             </td><td>19.0             </td><td>1999.0           </td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Group flights by month\n",
    "grouped = data.group_by(\"Month\")\n",
    "bpd = grouped.count().sum(\"Cancelled\").frame\n",
    "bpd.show()\n",
    "bpd.describe()\n",
    "bpd.dim\n",
    "\n",
    "# Convert columns to factors\n",
    "data[\"Year\"]= data[\"Year\"].asfactor()\n",
    "data[\"Month\"] = data[\"Month\"].asfactor()\n",
    "data[\"DayOfWeek\"] = data[\"DayOfWeek\"].asfactor()\n",
    "data[\"Cancelled\"] = data[\"Cancelled\"].asfactor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Parse Progress: [##################################################] 100%\n",
      "\n",
      "glm Model Build Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEPCAYAAAB/WNKuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXl8VNX9//88WSYzSQgEG8GiEgRRUKjQgrbWEqrgUhfc\ni5XUmlqrxaWl36K2Vas/Pi22tR9xoyr9YFgk2loVP1XQltSC1lAVtSa0tjVUUJmo/eCGrO/fH+fc\nzJ3JTDJJZsvk/Xw87iMzd+5y7p3Jed33eS/HiAiKoiiKkmoKst0ARVEUJT9RgVEURVHSggqMoiiK\nkhZUYBRFUZS0oAKjKIqipAUVGEVRFCUtZF1gjDEDjTEPGGNajDGvGGOONMZUGmNWG2P+ZoxZZYwZ\n6Nv+amPMq2776dlsu6IoipKYrAsMcAvwOxEZA3wK2AhcBTwpIocAfwCuBjDGjAXOAcYAJwJ3GGNM\nVlqtKIqidEpWBcYYUwEcIyL/AyAiu0VkG3AacK/b7F5ghnt9KrDCbdcKvApMzmyrFUVRlGTItgUz\nAnjbGPM/xpjnjTF3GWNKgSEishVARN4C9nXbDwNe9+2/xa1TFEVRcoxsC0wRMBG4XUQmAh9ih8di\n69doPRtFUZQ+RlGWz78ZeF1E/uLe/wYrMFuNMUNEZKsxZigQdp9vAQ7w7b+/W9cBY4yKkqIoSg8Q\nkZT4trNqwbhhsNeNMaPdqmOBV4BHgAvcuq8CD7vXjwBfNsYEjDEjgFFAUyfHz6nluuuuy3obtE35\n1S5tk7Yp1UsqybYFA3A5sMwYUwz8C/gaUAjcb4y5ENiEjRxDRJqNMfcDzcAu4FJJ9R1RFEVRUkLW\nBUZEXgQmxfnouATb/xj4cVobpSiKovSabDv5+xU1NTXZbkIHtE3Jk4vt0jYlh7YpO5h8HWEyxujo\nmaIoSjcxxiD54ORXFEVR8hcVGEVRFCUtqMAoiqIoaUEFRlEURUkLKjCKoihKWlCBURRFUdKCCoyi\nKIqSFlRgFEVRlLSgAqMoiqKkBRUYRVEUJS2owCiKoihpQQVGURRFSQsqMIqiKEpaUIFRFEVR0oIK\njKIoipIWVGAURVGUtKACoyhKSmlra2P9+vW0tbVluylKllGBURQlZdx3XwPDhx/KtGnfZPjwQ7nv\nvoZsN0nJIjplsqIoKaGtrY3hww9l+/Y1wHjgJUKhqWzatJGqqqpsN09JEp0yWVGUnKO1tZVAoBor\nLgDjKS4eTmtra/YapWQVFRhFUVJCdXU1O3e2Ai+5NS+xa9cmqqurs9coJatkXWCMMa3GmBeNMS8Y\nY5rcukpjzGpjzN+MMauMMQN9219tjHnVGNNijJmevZYriuKnqqqKurrzgaOA0cBR1NWdr8Nj/Zis\n+2CMMf8CPi0i//Gtmw+8IyI3GWPmApUicpUxZiywDJgE7A88CRwcz9miPhhFySwRH8xvgDLgQ0Kh\nM9UH08fINx+MoWM7TgPuda/vBWa416cCK0Rkt4i0Aq8CkzPRSEVROifig6nBPgPWqA+mn1OU7QYA\nAjxhjNkD/FJE7gGGiMhWABF5yxizr9t2GPCMb98tbp2iKFkm4oNpxLNg1AfTv8kFgTlaRN40xlQB\nq40xf8OKjp8ejXVdf/317a9ramqoqanpaRsVRekC64OZxW23nQhUAW3U1X1Dh8dynMbGRhobG9Ny\n7Kz7YPwYY64DPgC+DtSIyFZjzFBgjYiMMcZcBYiIzHfbPw5cJyLPxjmW+mAUJYO0tbUxZMhwRILA\nCOA1jPmYrVs3qcj0IfLGB2OMKTXGlLvXZcB04GXgEeACt9lXgYfd60eALxtjAsaYEcAooCmjjVYU\nJS4PP/wwtl9qBJ4DGhExPPzww53vqOQt2R4iGwL81hgjri3LRGS1MeYvwP3GmAuBTcA5ACLSbIy5\nH2gGdgGXqpmiKLnBxo0bgf3wJ1rCfm690h/JqSGyVKJDZIqSWdatW8fnPz8N+DNeqRg4irVrn+Do\no4/ObuOUpEnlEJkKjKIoKeP4409i9epGbHDnFqZPr2HVqt9luVVKd1CBSQIVGEXJDuvWrWP16tVM\nnz5dLZc+iApMEqjAKIqidJ+8iSJTFCU30EnClHSgAqMoeUpLSwv33nsvLS0tnW7X3UnCVIyUZNEh\nMkXJQy677Epuu+0u4ADgdWbPvohbb72lw3bJThLW1tZGa2srzz+/gW9/+yoCAVsWZtGiO5g589xM\nXZaSAdQHkwQqMEp/paWlhbFjP01suHBz83OMGTMmatv169czZUod27e/1L4uFBrHH//4KyZNmgRY\nC6eu7lKKiobz/vsbgeuB76EzVuYn6oNRFCUhTz75JHY2C3/C4zC3Ppry8nK2b/8H/knCtm//J+Xl\n5YC1XOrqLmX79jW8//7zWNGaD7ShM1YqXaECoyh5xpAhQ4DN+EUDtrj10XzwwQeEQkOBqcBEYCrB\n4BA++OADIP40yFa8WtEZK5WuyHapGEVRUszUqVMpKBD27j0KKwabKSgQpk6d2mFbKw7bgMgkYcac\n2S4a0dMge8Ntr1JefiF79rzBokV36PCYkhC1YBQlz6iqqmLp0sWUlBQTDO6gpKSYpUsXxxWCqqoq\nFi26g2DwdMrKzicYPD1KNLzPQ6GpVFRMJBSaysKFt/CHP/yKTZs2qoNf6RS1YBQlD5k581yOO+6L\ntLa2Ul1d3aWVYUwBEHJ/e3csRfHQKDJF6cckG6as9B80ikxRlJQQz4mvkWFKqlCBUZR+TLQTH1Id\nGZZsNQElP1GBUZR+TDwnfqoiwy677ErGjv00F1zwX4wd+2kuu+yKFLRY6UuoD0ZRlPZSMKly4nen\nmoCSW6gPRlGULulOUcqqqiomTZqUMsd+U1MTtg5aMXCv+7u/W6/0F1RgFCUP6W6F5FQzefJk4DXg\n08B/ub+vufVKf0EFRlHyDH/9sG3bnmP79jXU1V2ahfL6Rdghsr+5v8UZPr+SbVRgFCXPyIXQ48gQ\nWXQNMx0i61+owChKnpHu0ONksENhrxNdcHOzDpH1M1RgFCXPSGfocbKMGTOG2bMvAo4CRgNHMXv2\nRRpB1s/IiTBlYwsg/QXYLCKnGmMqgQZgOLYu+Dkiss1tezVwIbAbuEJEVic4poYpK/2aVIce94SW\nlhaampqYPHmyiksfIe9mtDTGfBsbZlLhBGY+8I6I3GSMmQtUishVxpixwDJgErYO+ZPAwfGURAVG\n6Sv0pU44F0RLSS95lQdjjNkfOAm4x7f6NGzwPO7vDPf6VGCFiOwWkVbgVUAHdZU+S1/Kds926LPS\n98i6BWOMeQCYBwwE5jgL5j8iUunb5l0RGWyMuRV4RkSWu/X3AL8TkQfjHFctGCWn6UvZ7lp1uf+Q\nSgsmq/PBGGO+BGwVkQ3GmJpONu2RUlx//fXtr2tqaqip6ewUipJZOgvl7anApGu4zQt93r69Y+iz\nCkzfprGxkcbGxvQcXESytmBTfP8N/At4E/gAWAK0AEPcNkOBFvf6KmCub//HgSMTHFsUJVdobm6W\nxYsXS3Nzc9Q6CAm8KCDubyhqm+4we/YV7nijBUIye/blqWq+hMNhCYUGR7U1FBos4XA4ZedQcgPX\nd6amj0/VgXrdEJgCPOJe3+QJCTAX+Il7PRZ4AQgAI4B/4Ib54hwvJTdbUXpLZx3/7NmXu88O7pUo\npFqs4rF8+QoJhQZLRcUECYUGy/LlK1J2bCV3SKXAZN0H42GMmULEBzMYuB87frAJG6b8f267q4E6\nYBcapqzkOMn4WVIxrHXvvfdywQX/hS3L4jGaxYu/z1e/+tXeXYQPjSLLf/LGB+NHRP4I/NG9fhc4\nLsF2PwZ+nMGmKUqPScbPMmbMmF77S6Iz5z0hS33mfFVVlQqLkjRZD1NWlHwmUyVTNHNeyUVyZogs\n1egQmZIrXHbZFdx2293Y3ODNnH32qdx++61psQT6UtKmkpvkXSZ/OlCBUXKJlpYWbrnlVhYvXkYw\nOIqdO1tZtOgOZs48N9tNU5QoVGCSQAVGySU0UVHpK+RVqRhF6Q/kwhwtipJpVGAUJQPkwhwtipJp\nVGAUJQNUVVVRV3c+/iivurrzkxoeW7ZsGaeddhrLli1LdzMVJaWoD0ZRMkDEB/MboAz4kFDozC59\nMAcccBCbN7+FF4F2wAFD+Pe/X+v0PJoIqfQG9cEoSh8j4oOpwU5nVNOlD2bZsmVOXP4M/B34M6+/\nvjWhJaPl9JVcQwVGUTJAT3ww999/P9Zy8VcBGObWR9PW1kZd3aVs376GbdueY/v2NdTVXUpbW1sq\nL0NRuoUKjKJkgKqqKn7xi59QUnIM5eXjCIWmsmjRHZ0OY51zzjnAZqKrAGxx66PRKDUlF1EfjKJk\ngPvua6Cu7lKKioazc+dr3HLLTVx88UVd7nfggSN4/fWtwDBgS0IfjObZKKlCEy2TQAVGyRV62/kv\nW7aM+++/n3POOYevfOUrCbfzRKy4eDi7dm3SSgFKj1CBSQIVGCVXWL9+PdOmfZNt255rX1dRMZEn\nn/wlkyZNSum5NIosu2zevJlXX32VqVOnZrspPUajyBSlD5HJJMuqqiomTZqk4pJBPvjgA+rr65k2\nbRrjx4/niSeeyHaTcoacmQ9GUfKVqqoqFi26g7q6qVHDVyoCfZc9e/awZs0a6uvreeSRRzjmmGP4\nxje+wSmnnEIwGMx283IGHSJTlAyhw1d9n1deeYX6+nqWLVvG0KFDmTVrFjNnzmTffffNdtNSRsZ9\nMMaYAHCgiPwjFSfNBCowSqp49NFHeeihh5gxYwYnn3xytpujZJitW7dy3333sWTJEsLhMOeffz6z\nZs1i7Nix2W5aWsiowBhjvgTcDAREZIQx5gjgOhE5PRUNSBcqMEoqGDduIn/960a8Ui3jxh3CSy+9\nkO1mKWlm+/btPPLIIyxZsoS1a9dy2mmnUVtbS01NDYWFhdluXlrJtJP/BuBI4P8ARGQDMCoVJ1eU\nXObRRx914hIp1fLyy3/j0UcfzXLLlHSwd+9e/vSnP3HRRRcxbNgwFi1axJe//GW2bNnCvffey7HH\nHpv34pJqknHy7xKR/zMmStDUNFDymra2Nu6+25vmOLpUy0MPPaRDZXnEq6++ypIlS1iyZAllZWV8\n9atf5eWXX2bYsGHZblqfJxmBaTHGnAMUGGNGAJdjH+kUJS/xEhZhIPAWNrx4PF6plhkzZmSzeUoK\nePfdd2loaKC+vp7XXnuN8847jwcffJAjjjiCmIdppRck44MpA64FpgMGWAX8SEQ+Sn/zeo76YJSe\n0DHr/lDg33ilWtQH03fZuXMnv/vd71iyZAm///3vOeGEE6itrWX69OkUFWnGhkfeZPIbY0qAp4CA\nWx4WkWuMMZVAAzAcaAXOEZFtbp+rgQuB3cAVIrI6wbFVYPKUdIb7xsu6D4UOYtq0cVx00UU6NNbH\nEBHWr19PfX09DQ0NjB07ltraWs466ywGDhyY7eblJJmOIpsIXAVU4xtSE5GJKWmAMaUi8pExphBY\nB8wBTgXeEZGbjDFzgUoRucoYMxZYhp1QY3/gSeDgeEqiApOfeMNXgYDNjk91vS0tGpkfbNq0iaVL\nl1JfX4+IUFtby/nnn69TVCdBKgUGEel0Af4GnAEcDIz0lq726+4ClAJNwFhgIzDErR8KbHSvrwLm\n+vZ5DDgywfFEyS/C4bCEQoMFXhQQgRclFBos4XA4pedZvnyFhEKDpaJigoRCg2X58hUpPb6SHrZt\n2yaLFi2SKVOmyD777COXXnqpPPPMM7J3795sN61P4frOlPTryQw8viMiD6ZEzeJgjCkAnsMK10IR\naTbGDBGRrU4l3jLGeGmyw4BnfLtvceuUfoA358n27R3nPIm1LnozjDZz5rkcd9wX2/cHO3SmGfi5\nx+7du3niiSeor6/nscceY+rUqVxxxRWcdNJJlJSUZLt5/Z5kBOZHxphfYoejdngrReSRVDRARPYC\nE4wxFcAqY0wNHcOgezTWdf3117e/rqmpoaampmeNVHKC6KKRdvgqXtHIVAyjVVVVUVVVlfYhOaX7\niAgvvvgiS5YsYfny5QwfPpza2lpuu+029tlnn2w3r8/R2NhIY2NjWo6djA+mHhgHNAN73WoRkdqU\nN8aYHwLbgTqgRkS2GmOGAmtEZIwx5ip37vlu+8exVQWejXMs6eralL5HV3OepNKHov6Y3OKNN95g\n+fLl1NfX89577zFr1ixmzZrF6NGjs920vCKVPphkLJgjReSQVJwsFmPMJ7CJnNuMMSFgGvAj4BHg\nAmA+8FXgYbfLI8AyY8wvsENjo7B+G6WfEDt8FdvRd2cYrStSeSylZ3z44Yc89NBD1NfXs379es44\n4wxuvfVWjjnmGAoKdLaRXCcZgXnWGHOIiPwtDeffD7jX2MymAmCJiPzeGPMCcL8x5kJgE3AOgPPP\n3I+1pnYBl6qZ0v/whq/ikewwWjKk8lhK8uzdu5fGxkbq6+t5+OGH+dznPsfXvvY1HnroIUKhULab\np3SDZIbIXgZGA//A+mAMdpgqJWHK6UKHyPovqZw6WKchzhwtLS3tpfD32WcfamtrmTlzJkOHDs12\n0/oVmc6DGRlvvYj8MxUNSBcqMP2bVCZj6jwu6aOtrY0VK1ZQX1/Pli1b2kvhjxs3LttN67dkRGCM\nMWUi8qGL7uqAiLyXigakCxUYRclNPv74Yx599FHq6+t56qmnOOWUU5g1a5ZWK84RMiUwj4nIicaY\n17Fhwsb/V0QOTEUD0oUKTP9BLYzcR0R4+umnqa+v59e//jUTJkygtraW008/nQEDBmS7eYqPTEWR\nXQQgIgek4kSKkg5i81SuuWYOF198UVyhUSHKPP/85z/bS+GXlJRQW1vLhg0bOOAA7Vb6A51ZMM/n\nuiO/M9SCyX/i5anAZwkGA/zqVwujnPGaMJk5/vOf//DAAw9QX1/P3//+d2bOnEltbS0TJ07UUvh9\ngEwNkb0gIhNScZJsoAKT/8SrfAwTge8SCl3WnhCpCZPpZ9euXTz++OPU19ezevVqjj/+eGprazn+\n+OMpLi7OdvOUbpCpIbJhxpibE30oIt9JRQMUpafEy1OxaVPTohIiIwmT+wHrgWpNmEwBIsJzzz1H\nfX09K1as4JBDDqG2tpa7776bQYMGZbt5Sg7QmcBsB17JVEMUpbtUVVWxaNEd1NVNZfv2SuBd4E7g\nzaiEyOrqaj766FXgEGAE8Brbt+/ShMke8vrrr7eXwt+1axezZs3imWeeYeTIuBkNSj+mM4F5R0QW\nZawlitIDvNIxv/zl3cyb91MCgfntCZF+68QW7W4kYukck50G91Hef/99HnzwQerr69mwYQNnn302\nixYt4rOf/az6VZSEdOaDWS8ikzLcnpShPpj8JVE0WKL18X01o7jxxgv5wQ+uyWDL+xZ79uzhySef\nZMmSJTz66KNMmTKF2tpavvSlLxEMBrPdPCVNZMrJ/yk6KZMvIi+logHpQgUmP+lONFhLSwtNTU2M\nGjWKadNOjYk2qyEYFP7977+rHyaGl19+ub1ky/77709tbS3nnnuu3qd+QqYE5k+d7Cci8oVUNCBd\nqMDkH51Fg7399ts0NTUxefJkxowZw2WXXcltt90FHAC8zpFHTuDZZzdg/TCbgDuoqJjPk0/+kkmT\n+qyhnjLeeuut9lL47777bnvJljFjxmS7aUqGyUgUmYjoILWSUyQqn/+tb13OAw88jCcm5513BsuX\nPwj8GU+Inn32KAKBQnbu/C52Vog3+31l5I8++oiHH36YJUuW8MwzzzBjxgx+8YtfMGXKFC2Fr6SE\nZIpdhoArgOEicokxZhRwsIg8lokG9hS1YPKPeBZMSckx7NixC7+YwGew0WL+GSZGc/HFx1Ff39Cv\nKyPv3buXp556iiVLlvDggw9y5JFHUltby4wZMygtLQW04kF/J5UWDCLS6QLcB1wD/NW9LwVe6Gq/\nbC/20pS+SDgclqamJgmHwx0+W758hYRCg6WiYoKEQoPl4osvERgtIL7lQIGQwIvu/YsCIWlubu70\n2PnMxo0b5fvf/74ceOCBMn78ePnZz34mb7zxRoftvPs7cOBECYUGy/LlK7LQWiWbuL4zNf1wlxvA\nX9zfF3zrNqSqAelaVGD6JrEd3MKFd3UQBL9INDc3xxWT8847360/WCAks2dfnsWryg5vv/223Hbb\nbTJ58mQZOnSozJkzRzZs2JBw+3A4LKHQ4Kh7GQoN7ndi3N/JtMA8DQSB5937EUBTqhqQrkUFpm/Q\n3NwsixcvbrcugsFBAssEwgLzBUIyYMCETp+mZ8++PK6Y+I8t0rlllC98/PHH8pvf/EZmzJghAwcO\nlJkzZ8pjjz0mu3bt6nLfpqYmGThwYpQ1WFExQZqamjLQciVXyLTAnIjNUAsD9wKtwLGpakC6FhWY\n3Gf27CucMIwWCMmRR35OoFRgosAggYFJP03Hikks+Tz0s3fvXnn66aflkksukX322UdqamrkV7/6\nlWzbtq1bx1ELRhHJsMDY81EFnAbMAPZN1cnTuajA5DYdh7bWxLxfJjAqJU/T+dpx/utf/5IbbrhB\nRo0aJYcccojMmzdPWltbO2zXHctt+fIVUlJSIcHggVJSUpFXQqwkRyoFpstYRGPMg0ANsFpEHhKR\ncHcDCRQllqamJmxY8XigDZubsr97DzaU+A1sVBjAS+zY8Rrl5eXdPpcX3hw59vj2Ypd9jW3btnHP\nPffwhS98gcmTJxMOh1m+fDktLS1cc801DB8+PGr7++5rYPjwQ5k27ZsMH34o993X0Onxn376GXbs\n2MXHHwfZsWMXTz/9dDovR8l3ulIg4FjgLuDfwAqsFRNIlcKla0EtmJwmYsHMFxgscFgHZ31RUZmE\nQoMlFDpcICSh0IgeDW/1dQtm586d8uijj8o555wjAwcOlDPPPFMeeugh2bFjR6f7dfe6EwVMJBp2\nVPITMj1EZs9JEdYf82vg/1LVgHQtKjC5iX+4pq7u6zEd2nz3fqRAqRQXl8tPf/pzKSmpcENoPReH\n2PDmXB/62bt3rzz33HNy5ZVXypAhQ+Szn/2s3HnnnfLOO+8kfYzuOu0XL14cJ+T7YFm8eHH7NmvX\nrpVrr71W1q5dm1QbuvKNKblHxgUGKAHOABqwTv47U9WAdC0qMLnH8uUrJBgcJGVlh0gwOEhuvHGe\nlJcfEdOhVQvc4KLIXpSSkgoZMGBCynwxuR5FtnnzZpk/f74cdthhMmLECLn22mvl73//e9Q2yV5H\ndy2YtWvXxrVgPDGZNu3EqGi96dNP7PT8sUEc/TFUvC+SUYEBlmMHyO/BDowXpuzkdtD9D9h5Z14G\nLnfrK4HV2FTsVcBA3z5XA68CLcD0To6dynuuOLr7BOvR3NwshYWlLjJsokBl+xBYdIdW6sTF/jrL\nyw+XkpJBfXZ4Kxnef/99qa+vl+OOO04qKyvloosukj/96U+yd+/eDtt2NxquO5ZbU1OTFBfvGyUi\nRUVV0tTU1KX4xKLDbX2XTAvMl4DiVJ0w5thDgSPc63InKIcC84HvufVzgZ+412OBF9xwXTXwD1y5\nmzjHTt0dV0Sk+0+wHgsX3iXFxeXOOqkUWNEuJpdf7j3ljne+mLIOYrJw4V19angrGXbv3i1PPPGE\nzJo1SwYOHCgnn3yyNDQ0yEcffZRwn576krpj8QQCAwV+K7BY4LcSCAyUcDgs1157rdgKCYsF1go0\nCVgLKx7JDLcpuUkqBaazCce8IID/NcYcaowZi0249NYv72rfJI79FvCWe/2BMaYFa9WcBkxxm92L\nzcO5CjgVWCEiu4FWY8yrwGTg2d62RemcdevW8cQTjfhrfq1efRTr1q3j6KOP7rC9V8/qj3/8E//v\n//0AGA28jv0aLwU2AvvwzjtvU1q6Px99dA/2meEPwGcpKxvF3r2b2+uFnXHGjLyoj/XKK6+wZMkS\nli5dytChQ5k1axY/+9nP2HfffbvcN1Gxz66mfq6qqkr6nonsAS7Efhet7j389a8t2Gi/a4D/AMOB\ntzCmMO5xRo0ahf2+/dNZb3brlX5DVwoE/AB4CptoucT9fTBVCuc7TzXWv1MO/Cfms3fd31uB83zr\n7wHOSHC8FGh5/8bvoLVPsAfHPJGOivsE6/laSkpGxBkmGSxwuMAJUePzcF77NiUlFbJgwYK8GU7Z\nunWr/Pd//7dMnDhRhg0bJnPnzpW//vWv3T5OuqPhIkEBYWehhKWiYoI0NDS472iN+/66Pn9nw21K\nbkMmLRjgXOAIbKmYWcaY/YDFqRA3D2NMOTY67QqxlozEbBL7Pimuv/769tc1NTXU1NT0tIn9jpNP\nPo3//d/VwL5AmDPPPAXYTPQT6RamT58etV9bWxuzZtWxZ08ACADD8OefWAN1I/BPoisgH4V9Kt7K\nnj0BfvjDxcyde31OVzzurOrw9u3bWblyJfX19axdu5bTTjuNm266iZqaGgoL4z/1d0VVVRWLFt1B\nXd3UqIrQqbLoqqvtJG7wJjAJeIlduzaxdetWbM5SGNgH2M/tMZ7CwgO47bbb+PKXvxw1d0x1dTVF\nRbvZtWs5sA0YSHFxXb+eHiFXaWxspLGxMT0H70qBcHXHgOeAAYABNqZK4bD+lMex4uKtawGGSMRP\n0+JeXwXM9W33OHBkguOmSM/7H6NGHSpQ4vwlE93fgHzuc593T6SjEvpg7NNuqXvKDbt9I2P6EJKT\nTjo57vg8zBFbIibxE3JvI8FSFUnmOc/Lyj7V7hfas2ePPPXUU/L1r39dKisrZdq0aVJfXy/vv/9+\nr84Vi3cNK1eu7FHARWfECwqwDvtiiYSQ+/1oIYGDJF6UWF8LDVcsZNjJ/0tgEPAtrBN+PVCfsgZA\nPXBzzLr5npAQ38kfwBbdVCd/ilm5cqVAkU8kvKGtUlmwYEGnUWThcFjmzJkj1pnvCUd0YMCUKccm\njDCCBidoEeHxhyT3tp5Ysvt3JUIRZ7jX/kekoKBEDjzwQDnssMNk/vz5snnz5k7b0luh62nARTLE\nti3+91UqEBSbu7RW4GKBQIdhzb4QGq5EkzGBcdbKfr73o4CJKTs5HA3sATY44XgeOAEYDDzpBG01\nMMi3z9WKWTXHAAAgAElEQVROWDRMOQ3U1dUJDHFPqn4LY6Q0NDQk3M/rvL2se9vxJA5Vja2AbH0w\n4YQWTG/9D8nun4wIrVq1yonoHQKfFdhXYKDcfvvtcUOLE90rv/XTHSIhw2vE+krWSGchw70lfkTY\nKIGqDg8Qhx02Pi1tUDJHpi2Yv6bqZJlcVGB6hrVgggLlUZ1xQUFZEk/00R0efFI6BgYcLNdee600\nNzdLQ0ODLFiwQH76059LKDRYgsERAgVih+dGCATlyCM/2/4UHAqNizpWKHR40k7jZLLauxKhHTt2\nyEMPPSQTJ04UQOB4gf8V+ItASO6+++4u29HR+nmxPRQ4WWzAxX7uOxri/g5NGDLcWxJbnEVx16dL\n6JTMkGmBWQpMSNUJM7WowPScceOOEAi4YZCRYkxpp0/Z9ol+P7ERRhPd3yEC18TtgEpKvKiiT7RX\n7G1ubpaCgpBEcmLKBAYIjJRgsFKuu+5HCa2hZEjGgoknQnCQXHzxN+Vb3/qWfOITn5AvfOELMnPm\nTHd9gwUmuL9DZcGCBZ22Ye3atfKVr3xFbD6J/xwjZdWqVcl9OeIfxvRbgIWycuXKpI/RXeLNuTNp\n0qQ4DxDxIwuVvkNGBAYocn9fAXa74arnvaGsVDUgXYsKTO9YuXKlnHfeefLzn/+8Syd7JIw1nk/l\nPPEHBvjDkb1x/IKCYILggNjjeWGvhwsMlmCwulthr105naNFqFVgtkCBGFMgc+fOlX/9618i4n+i\nXyN+i60zsYv1mcTeh+4IzN133x33fidjQfWG2Lpi3c3uV/oGmRIYbwbLkfGWVDUgXYsKTPdIpgSM\nzW+plFDocAkGK9s7aGvBRM/dAiOlpKTadUDfEbhW4JCYbSaInfclJNdcc43vGE1iLaFIPoa1arzO\nvEJgsZSUVHQ7V6Yzp/O2bdtkxozTxQ7TFTgroUhgQAchSzSLZqJ7G1+ADxWolOLi8m4NkV1yySVx\nLYdLLrkk+RuRIqZP94QzcWSh0rfIlMC8kKqTZGNRgeka74n0c587Jqqz9HcSXofc3NwsRUUDop7a\nCwvL2yPLYv0KxcU2WXLAAM9vEpbYJD37PiwwSqZOnSqFhZ7fJyzWr+APky6XSI2y/cSW7x/X6/DX\nXbt2yWOPPSYzZ86UgQMHyqhRB4sNyV0sXsFNCMUdfkq2UrCNrOsoCHYYMNjtIpDZsmAS0dP6dEpu\nkimB2Qx8J9GSqgaka1GB6ZxIpdv943ZWS5culYsvvsRVMx4nhYVlrkMcJHaIqlKsH2JouyjZyKjx\nEgoNluuu+5HMmTNHAgF/sIC/HH9sLsUwZy2Uu+PHtqnCdfhrOnzWk2z2DRs2yHe+8x0ZOnSoHHnk\nkXL77bfLxo0bpbAwKDbAwLOcrBjE8yskG4KbSBCsj2pNt9sfDoclkpfiDT0WayiwkhIyJTBvAtcC\n18VbUtWAdC39WWBiO77Y93bIpsR11l4JGP9w1CjXgR0sEWf/WImEH8d2kpGx96amJpky5ViJ9jcU\nS1nZeHeca5xQlThxC4odKhosMM5tf7JAbBn/kVJWNlpKSio6RJMlW75/y5Yt8tOf/lTGjRsnw4cP\nlx/84AeycePG9s9teZP9XBs+5dr0Q4GSDhZMV+HMfusmHA6LMSUx96SkXcDKyz/VHlmXLMuXr5BA\noFwCgSESCJRrEqOSMjLqg+mrS38VmNgci9mzL4/qCGfPvtwNZx3sOtAfSbysfZt1f1dMZ+vNPuk9\n2Y8UqHXvD2gfLov3tH7zzTdLMDjIHbdEbNn+w93fQJyn+45WyqpVq6S5ublb+TAffPCBLF26VI4/\n/niprKyUCy+8UBobG2XPnj0dtk0cjlstJSWD2jvxriLS4s2D4glCYWGVu95YoT5IIChnn31ut3J7\nUpXEqAmRiof6YFRgovD7SWJ9IZFoJ3F/S1wn730+SGJL5Nt9miU26dGKy1ixlo4XBXakEyUbTnz6\n6WdKotBVr5ONVyXAPweMFa46gZAEg4d1sBC6igbbs2eP/P73v5cLLrhABg0aJCeeeKLcd9998uGH\nH3Z6H5uamqSo6NCYto9vv15PRDrLyUkkUl6uT1nZp6S4uMLNhXOwRLLhV0Tdx2QsklSXvelphQQl\nv8iUwAxO1UmysfQXgfF3DnZirqEdOnfbQa5wAnGwWH/GmWJLfOwv8bL27aySE6RjZ1siNot9kHsS\nL47qTK2AxA9dbW5udhFQHSPObDSZX3Bstvy8efPidqDxOtfm5ma5+uqr5YADDpAjjjhCbr75Znnz\nzTeTvpfxkiD9Fps3FNfZZFqJst7tfDjRvqhAYKy71rsk2SrF8b733ohCuis0K32PjCZa9tWlPwhM\nvM4h2mLx3v+2QwcWqSVVKMXFFXGOUSIdLQ0ve3sf8bL9S0oOiupMKyomyJFHfk5iQ1cjw0YHxe2c\n7VDZBIlEjK0RCMnChXd1eQ8WLFggn/nMZ2TIkCFy/vnnS2NjY/tn3X3Cv/HGeRIJRIj2OfktmOLi\nIRKvFH2iIcKyssMlcTTdAIkV8878SqkUhWQqHCj9CxUYFRgRkbhDNbZjLHNP0QMFiqSgoMR17P7t\nIjkohYWDJZJB780q+VuBS9wxvM62RGCB2MCAxRIMVjq/SseObuXKlVJXVycrV66M88Rvn+CDwcPc\n+jPFWlUjnHAdKJ7PJ17HuX37dnnggQfklFNOkYEDB8r5558vV111tQSDlT5f0xU9esIPh8Pumu4U\nG5AwSGBk1DEi17O4/V54Fky8eVAKC/fxTfvcJNan5f8uqjuIeWeCkawoJCOwasEosajAqMCISCKn\ntD9ayb4uLh7T4Wncn4NirRJvOtxmscNppT5hKXYCMM91uBPEm+44nj/EP3wTDFZKTc1U6eiXqZaD\nDhopdXUXScSy8YINvGi2sJSWjpZVq1bJ3r17Ze3atXLxxRfL4MGD5Ytf/KIsXrxY3nvvvTid5JoO\n96U7nab/moLBQXLjjdHDdE1NTVJQUCV+R74xg9o7c9uWyBQF/mmfy8s7hmAXF1dIbe0FEgxWJlXa\nPhlR6M4QmpbVV/yowKjAiIhnwYyQSE2sQb7OK1G5lWrpmIPyZekYLRZ0T+erxIbq+i2c2QJBmTdv\nnojYDm/VqlUJo7zssWKFsFKgxPdk79/2TrF+iUqB4VJUFJIhQ4bImDFj5Mc//rH8+9//7nAfop/o\nmyTWD9LdYZ/Onv47DoOtESiRk046RUTid9jePWpoaJCiojJ3bXZI0Mvk786QXmei0BOrRKPIFA8V\nmH4qMP5oseiosTVOCOZKZC4Wr9yK+JaRYi2WoEQS9ELS0ScQEmvB+HNTYj+vkoaGBgmHw3LVVddI\nIFAhAwZMkJKSQU70miViEQ11x/NbRXcJ7B8TteVZTgeItZqOEFsS/jwpKamQrVu3JrwvqbRgusI6\n8g/2tdkLnrDRYv7vKhwOt4vBgAETpLi4wuXaNIt/WuKe+DwSiUK8IbSysvGd1jtTgVE8VGD6ocBE\n5luxTnKvTEpBgVdav9QnGvMlngUTCAyUYHCQlJaOkoKCgNgosP2kY1SXl2g5V+yw2GExn4+TwsJS\nWbjwLhcgEJuZ72WZj5ZIYIBnVS1z6+6UyHDeiwJb3DVMdn8HCjwisFOSsUBin+i9/J90DPtEF7uM\nFmev9L7/YaCjRedd34qUi59IouCPUgkGB8W9D72dn0bJL1Rg+pnARI/rV7i//nlXBrqOO1I7yyYx\nlok/mmv27Mvbh2qsI3u+2GG1UolYQXe6fSLl+qP9N78VKJbvfvd7cTqxwWJDn+NZPM0SbUkVytCh\n+wkUOqHxCkwOE2thlUh3LZCuKhikkjPPPNu1Mdq3VFY2Xm68cV57UdBAYIAEg4dLtEB7ARaJO/3e\n4hUmjRX/eFNQ93Z+GiW/UIHpZwJjfS0HOTHwSo14vpcBTgi8eVhWiPWleMUaR4id6/4mCQTK24fX\n7JwsFU5YrpBIWLInKP7kyzUSyb6PCFZh4QFxOs6LO3S6dvsbfIJTItaqCQgYscNw0ROc2XPZCLZ4\nPoZMDeckKmhp64sVdBDT4uIKV7ct8XCdF2ARCh3erTL93SGS1Hqg+JNYYy3BRJWw09UuJfdRgeln\nAhM/WsyzaOI5zyvEWiOeMAxwnUipFBaWyXnnzZLIEJYXlhwbEOBl13s+hhFxzhWbc1MqkeGx2O1C\nAp8Qa7EgcJbAOoENro3xnvJXSUlJdGeXyazzeCVfPE466WR3z7zrtcI8YcJE6Zi4WuXbJrE1kSoi\nFu8aJ9KJrRMrMB0rK6jA9F9UYPqZwMTPd5kgNicl1loYKdY68MqQxHYe8TLtSwXGxDnOnRLxMcTL\n3xjpxMHz/QxxYuXNETJSIlaKNwTm5dz8XCJBAPHEy3aQ/k44kzkbnWXrh8NhKS4u833u+ZaCctZZ\nZ8Xd75xzviyFhZ7QjpRAYGDaxDHi5PdPexAdseZhr6XzbZT+hQpMPxGYzh3FA8RaKvGy7de493fG\nPE2H3dN0PKd+iXQswFji2zYssU/DkWTDCom2ZIJSUFAsoVCpGFMgNljA8w95T9b+rP4iMcYbfvOG\n6IZ06IQzmXUev+TLwbJ48WKXB+NFjkXfx0jwRKTDhoCsXbs2Kpy7q1lCe0NEiJdJ7MRt8e6X9dcM\nkrKy0WnzCSl9BxWYfiAwsUNBXlRUKHS4RFsmkSKJJSWDpLjYP9/73XG282aZ7GgtRPJkBogtB3ON\nRD+lR+dvWAFa5TqxNwVuFhtaXCTnnnuuNDQ0yIABE2I64ZFiI9MiRSTBTpbV3NzcnisS2wmLZNOC\niVgozc3NvukO4g0FXibWGoyEaQcCYxKKYLqG/DzRSLZCgIYpKx4qMHkuMIk6Us9B39DQIGVl/uGq\nsJSVjY4zsdVciUSDlbgOb7HYKYw9Z/1g1+GHXec/WOwQmxdKPNytG+22988bM0LgQrF+iAECFwjc\nLcXFFRIOh2XhQq/cf7wcm0HS1YRe8chk1rmdFjkS/FBUNECWL18hCxYscPcietpkOwS40t2LSvGm\nP0g05JRuwQyHw3LjjfM0S1/pFioweS4wXQ0FJeqY5s71BMWzMvxFK4fGdIaDxTrcvSKT3rwsFW59\npUQspTUCDW6/FwQaBWaIddYXinXeR5I3Fy68y9fGOrffWHfsH7pjTo0Sne5Mt5upp21blyw6+CEU\nGhwj5M1iI+S8KLu50pVj3SNTQ35qnSjdIa8EBlgEbAVe8q2rBFYDfwNWAQN9n10NvAq0ANM7OW5K\nbnY2iBRcjOS2xD7Z2qfriGDMnn25nHDCCRKZnXKVsxIqJV5CYER8YtcVOAHwqjBPlcjQWZn73Cv9\n8gknIE3iTSTmZbJHOs9VYp3/Xg2zwe59QDzn/vTpJ8a9B9nuFBMJwKpVq+LMUBkQCDrnf3JlarTQ\npJKL5JvAfB44IkZg5gPfc6/nAj9xr8cCLwBFQDXwD8AkOG5q7nYWsDkMA8ULLS4uLk9Qa2qNeAmX\nweAgmT17tkTySr7nOr+JEr9sjDc85g1TeYUvA1JZWSkRC+YT7pieIN0tkSRPb5tqKSgojyqtH2nj\nYok/TPZJOfPMM+NaLrkyAVaiwpVe+RevZI69BwGZPv1EaW5u7lBfrTPR0EKTSq6RVwJjr4fhMQKz\nERjiXg8FNrrXVwFzfds9BhyZ4JgpuNWZJ95TbTBYGeX07vhkvUKiEyXtPC+R6sRrJL4Fc4D7/HL3\n1ws3xlkqAwRmCtzhjukldXqWyAqxTv0bBILt1ot3HTfeOE+KijxfT6y4FSb0S8QblkpXvoi/tlu8\nc0yb5oVcW0vFs7Y6szK7Kxq5YK0pikd/EJh3Yz5/1/29FTjPt/4e4IwEx+z1jc4G8YZlbITYcCku\nLpOvf/0iaWho8M3D4uU6eJOGedFi1hqJOPm9CbI8ETnPJzZBgdvFzv8yyInLPgLbotpgs8KDEl2W\nxrOCDpbi4rKoDtZeR6JoqxJZunRph+s/++xzJTZRMR1+iUhtN1vMMxQa0UEMOsuFScZPpqKh9EVS\nKTBF9A2kJztdf/317a9ramqoqalJUXPSR3V1NTt3tgIvAePd363s2LEbOIB77lnKPffcT0HBTgoK\njsKYAHv2DACCwFvAN4H/AdZjRxB/CwSAt4EbgB3ACmAZ8BrwMLAHuAn4BrABOBbYDLT62vAu1iX2\nP8DXgU+6dXcCbwKbKS4exooVK5g793q2b1/D9u3FwKeB7wLHAPu7c54M/IWf/vSnTJ8+naqqKgBa\nWlp44IGHgVDU9e/c2Up1dXXK7nFbWxt1dZeyffua9nNs3z4V+A11dWdy3HFfpKqqiqamJuAAtw3u\n7/40NTVx0kkndfiedu3a1N7Oqqqq9utSlFymsbGRxsbG9Bw8VUrVm4WOFkwL0UNkLe517BDZ4+Th\nENmNN85z4/gjnfUR8D1Fr5HI0JdXk8wLK/Ymswr49vU+85zRJzsr5NNi/SvnuuP9wfeUXiE29LhS\nbL5KqbOMvKf1ajnssHHuON5xjxYISWmpF84sYv0Wn5To+Wr2bbdqiooOibIaIsmNK3z7lMo3v3lJ\nSu6rZ1HEtxInCDRFWSGdWTAiqfOfqLWj5BLk4RBZNfCy7/18T0iI7+QPACPIMye/f2ipuHiAb7hq\niERmmozMPRIpyx9bKLLSrV/jtvuLwKMCxwvtocVFTpAGix0uC0lk7pcrpGM0WsTRDSG5+eab5brr\nfiSFhUE3/0u8ic4SVVaOLgZZUlIhzc3NnSY3puq++meX7Ko0jUj8aD0/vRWHXAloUBSPvBIYYDnw\nBnbs5t/A17Bhyk9ix2RWA4N821/thCWvwpTjObejpz/2nOyxneIF0rG44gSBZwWWOHEYIvBZsYmT\n4yQyJ4sXKjxY7ARgTe5cXsl+71xHS7QVVCJQIgUFQTeJVnlMSfq7xM5WOcK1zZ+ceYhbxCeYw6Ww\n0E7W1VWH3pP7Gi8U2BMZWxkhJMFgdcIOPlFF5d6iYcpKLpJXApOupa8JTEfndliiy3wsiyMkIyV6\nmmQRWO3eHyo2ubJI7MRdflHysvavdWKxJsbCKBVr5QyQ+KVlPJEpEmtZrfFtEwkyCAQqpKCgVPxZ\n7dba8mbh9FtPtmLxhRdelNIOvTNnfDJRZOkkk7XVFCVZVGDyTGDs0FBQokvmxwpKfH+A7axtQUUr\nCMaJwyfd5/5y8pUSmXUyJDY737Ng/EmQxe44XptiizoeLDY0OSS2XplIMFgtgUBs8U2/8PjbXOwE\nanjcz1NpKeSylRAOh6WoKNoqLSoakBNtU/ovqRSYgt6HCSi9xUYrHYiNyJoKTMRGar2BjVJqAI4G\nKoCjgMOBI4Fd2Aiw/wB7gQuwkV9PAu8Bi4Et2BiKI4CPgf9y56gANgHbgN8Av3R/PwAexEaT7QI+\nxEaUveRa+5J7fxYwzJ37JYx5jx//+DpKSqqJRF2VERuFVVx8IFde+S3Wrv09hYXhDp97UVqpoqqq\nikWL7iAUmkpFxURCoaksWnRHTkR4vf322+zevQOowX7nNezevYO33347uw1TlFSRKqXKtYU+YsGE\nw2FpaGiQeM5t6wyPndPlVxKZtOtgsUmSxW7bUjekVSK25pdnFfgnn/ImIit1+9j8mEDgUImOFgtL\nMDhczjzzTKmp+aL4a43Zc1pro7R0lIRCg11CYmxb41swAwaMk1BosJx33ld6ZMH0xLGei5Fakag5\nv4/KTgmgKNkCHSLLD4HxIoiKiw+WjuHEXin8wU40/lsi0yIXiU16HCiRUOGwwNVifTKeEAyTyERg\nXvSWv9R+iVx11TXtZfIjyZsrfG2w1QEKC8tl2LADxT/B2JQpx0pTU5MrXx/rgxkpwWBl+zQDAwYc\n4baZLf7M9/POOz/qurty6udT1FVXYdCKkg1UYPJAYKJ9A95skZE5ROz7a3zWyikCDzgB8XdIg9yy\nRjqWgxkg1rqZKxF/zWjxKicHAp+McihH5hAJSkefkK0KvHTpUrn22mujaoh1nJwrLHCALFiwoP1a\nL774ErF+oiFiHf0r2h3ayTr1c9mf0lNSHTWnKL1FBSYPBCY6gijsxGGDwFMCZ4p1sn9GbL2wQWKT\nHw+SjnW9Joidz6U8zmfeTJWHOEvEC0UOi02gLOnQqa9atUqKiz/pztuxQGZJSUUHq6GrJ/FwOCzR\nIdfW0R8MDuqWOORr1FW6wqAVpSeowOSBwEQ/jb8qcKwTlSK3FEp07bD9ncjE5sJUOuFINH3yTRKp\nG1bhez1QoFrOPvvcKP9EOByWwsLSuBaM58uJN79JZ0/iHSdCs237+tcv6sU9kx5bMKn0x+Sib0dR\neoMKTB4IzDvvvCNnnHGm2Ez9QrcEnZUSr0Ck56gPxAhPscSbPjnieykVb0jKrj9IIhWRbXhzMFgZ\n5dO47rofOZHzCmV6pf29AICRsmrVqg7XlOhJfM6cORJv/vo5c+Z0+771tjxLKn04+eQPUhQPFZg+\nKjA7duyQhx56SM444wypqKiQyZMnO0tiSIy1cEOcDtnWyrIiMUIis0N6dce87ZqlY/JkpTuPN2e8\nN2HY9zoIWSg0WFatWiWFhcOcMA2Jc7yQNDQ0JH3dS5cujWvBxKumnAw9tRpS6cPJR3+QooikVmD6\nSjXlvOCEE05gz549zJo1i0WLFvHmm28yduxEYCcwGpsH0gaUE8k98aoZb8LmpHhVjd/E5rMMw+a6\nLAcmAL/GVjqucWcdjy31FnbLL7G5M58DbsHm3/jzVIYDUFT0IXv2rHTn/SFwJjafZhNQwaBBg5K+\n7t27dwMF2Bwer70Fbn338SoVt7W1sX79eqqrq5PKa2ltbSUQqGb79ujrbW1t7XZeTCqPpSh5S6qU\nKtcWctCC+fDDDzuss74LL9t+vrMavBIvQbG+l6DYkOPYqsbjJTpTP+T29Y7lWQsV7hjFEu3fWSOx\nkWfeU/jChXe5bca6fe8UL4cm0RzziYgEASwWW57GznLZG6d2T4an1IJRlK5Bh8j6psAkorm5Wc45\n58sxnb5Xo8sru++VV/EPM3nhxLG+mt+6bQ+VSBmZkMDZbvsiiQzBeQUnR0lR0YCojnrhwrucw7+0\nvR1FRWVy443zut2RpjIctzedeyqnKNbpjpV8RAUmzwRGxIu0GiXWzzImjs+iUiLZ/d60yDgREd9y\nuLNYvHyXu9z2nmWzj8/q8WfcF8uVV14ZZVXE68j9mfjd7VC7CsdNNly3t+HKGkWmKIlRgckzgZk9\n+wqJRGytcSISz8m/vxveKhOb3R+vhL9nBfmFKSTjxo2X4uIyKSoaIkVFpXLood7kZKN8gjM6yrqI\nPzHXeCeCqR0SsvcgOQtHh6cUJX2owOSRwEQqKd8p1v9SKZFKyIksGP96T5g8H8x+MYIwUqBQVq5c\nKQsX3iWBQIWUlo6WYHCQXHfdj+SSSy6Jcy7rH4lvwXjl/lOX5NiTkik6PKUo6UEFJo8E5rTTTncC\nMdEJzACfpeL3wQTculjLZrz7LODbx99Rl0p19UESDofdLJmRuVmKi8tlwYIFEl3mRcRfcNHryCO1\nxCLBA6myGubNmxfnukbJvHnzOt1Ph6cUJfWkUmC0XH8WaWtr49FHnwCeAZ4D1mHDecPAvsBu4G1g\nO7Ycfzk2xDe6dH5p6ShWrVrJ2rV/pKBgL/AFvPLvhYXQ1PRnXnjhBXbt2gM0unM1smvXXkKhEPB6\nh2NOnjwZgJkzz2XTpo38/vd3sXDhLYRC89NU9j52SoAtXe5RVVXFpEmTNCxYUXIUFZgs0traSmFh\ndB4KfAKYSkHBe3zlK+dSVLSToqLBQAB4CvgVNsdlFDAFmIvIW7z22iamTTuVkpLRwA4CgbcIBoUl\nS37l64A/GXOu/TjwwAOZPfsibI7KaOAoZs++iDFjxrS30+vIL774IjZt2siTT/6STZs2MnPmuSm5\nD6effjp27plIG2CXW68oSp8lVaZQri30gSGycDgsgcDADkNaEJTLL7+ifRs7jOUvZBkWGCbB4PCo\n+eX9xykpGdQhIiz2XP58lmwXXLRhzCUC+wqUaFVhRckSqA8mPwRGRKSm5tgYX0tICgpCUX6FeOJQ\nXFwhq1atavdDJBO26/lTysrG56RjPNsipyhKagVGS8VkkZaWFhobnwa+C/wCqAK2Mn/+vCi/QlVV\nFYsX/5K6uqkUFOzP3r2bWbToLqZPn96+zc6drfhLy+zatYnq6uqo8x1xxHjmz7+eIUOGMHXqVIBu\nlVpJN2PGjIkamlMUpW+jApNF7NzzBwA3AJcBrcC5VFXt02HbmTPP5bjjvkhra2sHQfDmna+rm0px\n8XB27doU5YBva2vjW9+6nAceeNid73WmTath7dpnCQSq2bmzlUWL7kiZT0VRFAXAWIso/zDGSK5f\nW0tLC2PHfhr4M5GilkfR3Pxcj57k29raOgjQffc1cOGF3+Tjj3dio9XGYyPJToo6byg0lU2bNuaE\nJaMoSvYwxiAiJhXH6pNRZMaYE4wxG40xfzfGzM12e3rKmDFjuozg6g6xYbttbW3U1V3Kxx/fDhxK\nJIKsDNif2CrKra2tPb8YRVGUGPqcwBhjCoDbgOOBw4CZxphDs9uq5PDKy7e1tbWvu/XWW2hufo7F\ni79Pc/NzXHvtDzps093jt7S0sH79el544QUCgWpgGnb4zcsz+ZDYvJN4PpveEHutc+bMYfjw4cyZ\nMydl50gn8b6reOv6G8ncg3Xr1nHdddexbt26DLYsN9DfSAypihbI1IJ93H/M9/4qYG6c7XocRZEO\nkikv35sZEr19Q6GDBEISCo2TYHCQL/osdrbLE8WrohwIDExpRFnsdURqndk6Y4WFgZSdKx3E+x50\n9srkfp/Tpp0Y9V1Pn35iFlqaHfLlN0J/DlPGznx1l+/9+cCCONv16iankmSKM/amgGNk3zUSO79L\ncXF5e82uQKDClZP5rUSqKJfI2rVr03qtHQtwhuQ73/lOys6ZShJ9D8HgoB59N/lCMr/PtWvXxilV\nFM7LsIEAAAomSURBVErp7ytXyacCrKkUmLyOIrv++uvbX9fU1FBTU5OVdiQz+2FvZkiM7FuGnb0y\ncoxQ6GAeeOAnVFZWUl1dzQ03/H/cdtt5WB/MZmbPvpijjz46rddqZ7Esi3r/61//mp///OcpO2+q\niNf+goL9gR3E81n1l6CIZH6fq1evJta3B8NYvXp1Sn9juUhfnuG0sbGRxsbG9Bw8VUqVqQU7RPa4\n733OD5Fl04KJd4x0JjSqBZOfqAXTOWrB5M8QWSHwD+wE8QFgAzAmzna9vM2pJZny8r0pQe/tGwxW\nOx/M4VkbB469DmOKJDL3TN/xwfi/B50eILnf5/TpJ0Z91/3RB9PXfyOpFJg+mQdjjDkBuAUbBbdI\nRH4SZxvJtWuLl6fSk226On55eTkffPBBVjP0Y69jzpw5/PrXv+ass87KyaGxWOJ9D735bvKFZO7B\nunXrWL16NdOnT8/7obFY8uE3kso8mD4pMMmQiwKjKIqS6/T7REtFURQl91GBURRFUdKCCoyiKIqS\nFlRgFEVRlLSgAqMoiqKkBRUYRVEUJS2owCiKoihpQQVGURRFSQsqMIqiKEpaUIFRFEVR0oIKjKIo\nipIWVGAURVGUtKACoyiKoqQFFRhFURQlLajAKIqiKGlBBUZRFEVJCyowiqIoSlpQgVEURVHSggqM\noiiKkhZUYBRFUZS0oAKjKIqipAUVGEVRFCUtZE1gjDFnGWP+aozZY4yZGPPZ1caYV40xLcaY6b71\nE40xLxlj/m6M+e/Mt1pRFEVJlmxaMC8DpwN/9K80xowBzgHGACcCdxhjjPv4TqBOREYDo40xx2ew\nvb2msbEx203ogLYpeXKxXdqm5NA2ZYesCYyI/E1EXgVMzEenAStEZLeItAKvApONMUOBASKy3m1X\nD8zIWINTQC7+oLRNyZOL7dI2JYe2KTvkog9mGPC67/0Wt24YsNm3frNbpyiKouQgRek8uDHmCWCI\nfxUgwPdFZGU6z60oiqJkFyMi2W2AMWuAOSLyvHt/FSAiMt+9fxy4DtgErBGRMW79l4EpInJJguNm\n98IURVH6KCIS67roEWm1YLqB/2IeAZYZY36BHQIbBTSJiBhjthljJgPrgVpgQaIDpuoGKYqiKD0j\nm2HKM4wxrwNHAY8aYx4DEJFm4H6gGfgdcKlEzKxvAYuAvwOvisjjmW+5oiiKkgxZHyJTFEVR8pNc\njCLrFsaY64wxm40xz7vlBN9nOZGwaYw5wRiz0Z1vbrrPF3PuVmPMi8aYF4wxTW5dpTFmtTHmb8aY\nVcaYgb7t496zFLRjkTFmqzHmJd+6brcjld9dgjZl9fdkjNnfGPMHY8wrxpiXjTGXu/VZu1dx2nSZ\nW5+1e2WMKTHGPOt+168YY/7Lrc/mfUrUpqz3UcaYAnfuR9z7zNwnEenTCzYA4Dtx1o8BXsD6maqB\nfxCx2J4FJrnXvwOOT2P7Cty5hwPFwAbg0Azen38BlTHr5gPfc6/nAj9xr8cmumcpaMfngSOAl3rT\njlR+dwnalNXfEzAUOMK9Lgf+BhyazXvVSZuyfa9K3d9C4M/A0Tnwm4rXpqz3UcC3gaXAI5n83+vz\nFowjnkM/VxI2J2P9RZtEZBewwrUtUxg6WqqnAfe61/cSuf5TiXPPUtEIEVkL/Kc37Uj1d5egTZDF\n35OIvCUiG9zrD4AWYH+yeK8StMnLQcvmvfrIvSzB/sb/Q/Z/U/HaBFm8T8aY/YGTgHtizp32+5Qv\nAjPbGLPBGHOPz9TLlYTN2HZkOkFUgCeMMeuNMV9364aIyFawnQewr1uf6J6li3272Y5MfXc58Xsy\nxlRjLaw/0/3vLC3t8rXpWbcqa/fKDfu8ALwFNIoNEMrqfUrQJsjub+oXwP/D9gUeGblPfUJgjDFP\nuLE/b3nZ/T0FuAM4SESOwH6pP89ua3OOo0VkIvYJ5lvGmGOI/qER5322yIV25MTvyRhTDvwauMJZ\nDVn/zuK0Kav3SkT2isgErIV3jDGmhizfp5g2fcEYM4Us3idjzJeArc4C7Sx1Iy33KVfyYDpFRKYl\nuendgFchYAtwgO+z/d26ROvTxRbgwAyeLwoRedP9bTPGPIQd8tpqjBkiIlud6Rv2tTWT96a77Uh7\n+0Skzfc2K78nY0wRtiNfIiIPu9VZvVfx2pQL98q14z1jzO+Az5AjvynXpv8FPiMi/oK+mb5PRwOn\nGmNOAkLAAGPMEuCtjNyn3jiOcmEBhvpefxtYLtHOqgAwgmhn1Z+xHa3BOqtOSGP7Cok4+QNYJ/+Y\nDN2bUqDcvS4D1gHTsQ6+uZLYwdfhnqWoPdXAy7733W5Hqr+7OG3K+u8JO759c8y6rN6rBG3K2r0C\nPgEMdK9DwFPAsdm8T520Keu/KXfMKUSc/Ddl4j6lpOPI5uJ++C9hO+6HsGOL3mdXuxvUAkz3rf80\ndrqAV4FbMtDGE7CRN68CV2Xw3oxw9+UFd71XufWDgSddm1YDg7q6Zyloy3LgDWAH8G/ga0Bld9uR\nyu8uQZuy+nvCPnHu8X1vz7vfT7e/s1S1q5M2Ze1eAeNcO14AXgS+29PfdgbalBN9FNECk5H7pImW\niqIoSlroE05+RVEUpe+hAqMoiqKkBRUYRVEUJS2owCiKoihpQQVGURRFSQsqMIqiKEpaUIFRlE4w\nxuxxZc7/6sqwf8f32ac7K1tujBlujJmZmZYqSu6heTCK0gnGmPdEpMK9/gRwH7BORK5PYt8aYI6I\nnJLWRipKjqIWjKIkiYi8DXwDmA1gjJlijFnpe/2Cs3aeM8aUAT8GPu/WXeEsmqeMMX9xy1G+fdcY\nYx5wkzwt8c5pjJlkjFnnKvH+2RhT5ir23mTs5FYbjDEXZf5uKErX9Ilil4qSK4jIa66Dr/JWub9z\ngEtF5BljTCnwMXAV1oI5FcAYEwSOE5GdxphRWGtoktv/CGwdqLeAdcaYzwHrsfMHnS0iz7tqxh8D\ndcD/iciRxpiA2361iGxK9/UrSndQgVGU7hOv7Pk64BfGmGXAgyKyxZgOmwWA24wxR2Brex3s+6xJ\nXOVrY8wGbBHO94A3ROR5aJ/sCzeN7ThjzNlu3wp3LBUYJadQgVGUbmCMOQjYLXb6g/b1IjLfGPMo\n8CWsRTE9zu7fBt4SkfHGmEJgu++zHb7Xe4j8b8YTMwNcJiJP9OJSFCXtqA9GUTqnvYN3w2J3Ard2\n2MiYg0TkFRG5CTu0dSjwPta68BgIvOle12KncuiMvwFDjTGfducod8K0CrjUzdGCMeZgY0yoJxen\nKOlELRhF6ZygMeZ57PDWLqBeRH4RZ7srjTFTsdbHK8BjWP/MHjeF7mLgduBBY0wt8DjwYYJzCoCI\n7DLGnIsdVgsBHwHHYedWrwaeN9aMCtOLOdsVJV1omLKiKIqSFnSITFEURUkLKjCKoihKWlCBURRF\nUdKCCoyiKIqSFlRgFEVRlLSgAqMoiqKkBRUYRVEUJS2owCiKoihp4f8H022p+JC2lWIAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e128630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate and plot travel time\n",
    "hour1 = data[\"CRSArrTime\"] / 100\n",
    "mins1 = data[\"CRSArrTime\"] % 100\n",
    "arrTime = hour1*60 + mins1\n",
    "\n",
    "hour2 = data[\"CRSDepTime\"] / 100\n",
    "mins2 = data[\"CRSDepTime\"] % 100\n",
    "depTime = hour2*60 + mins2\n",
    "\n",
    "data[\"TravelTime\"] = (arrTime-depTime > 0).ifelse((arrTime-depTime), h2o.H2OFrame([[None]] * data.nrow))\n",
    "\n",
    "scatter_plot(data, \"Distance\", \"TravelTime\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "glm Model Build Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEPCAYAAAB/WNKuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXt8VNW5978rydySkAQUQQEBuUhQUVCjVqtBhVbbao8e\nL8gLtaJVKYgtPYqet4hS2uKtr2ItFbFIkItttcppNWhLjkWrwbs1qRXb4LUmPVYtpyC35/1jrZ3Z\nM5lJJmRuSZ7v5zOfzOzs2XvNnmT99nNdRkRQFEVRlHRTkOsBKIqiKD0TFRhFURQlI6jAKIqiKBlB\nBUZRFEXJCCowiqIoSkZQgVEURVEyQs4FxhhTboz5uTGm0RjzujHmOGNMX2PMBmPMG8aYWmNMuW//\n64wxb7r9J+dy7IqiKEpyci4wwB3Ab0SkEjgS+BMwD3hSRA4FfgdcB2CMGQucD1QCZwB3G2NMTkat\nKIqitEtOBcYYUwZ8XkR+BiAiu0XkE+Bs4H632/3AV93zs4C1br8m4E2gKrujVhRFUVIh1xbMcODv\nxpifGWNeNMbcY4wpBgaIyIcAIvI34AC3/yDgHd/733PbFEVRlDwj1wJTBEwAfiwiE4D/xbrH4vvX\naD8bRVGUbkZRjs//LvCOiDzvXv8SKzAfGmMGiMiHxpiBQLP7/XvAEN/7B7ttbTDGqCgpiqLsAyKS\nlth2Ti0Y5wZ7xxgz2m06DXgdeBS42G37GvCIe/4ocKExJmiMGQ6MBOrbOX5ePW644Yacj0HH1LPG\npWPSMaX7kU5ybcEAXAU8YIwJAH8Bvg4UAg8aYy4BtmIzxxCRBmPMg0ADsAuYKem+IoqiKEpayLnA\niMgrwLEJfnV6kv1/APwgo4NSFEVRukyug/y9iurq6lwPoQ06ptTJx3HpmFJDx5QbTE/1MBlj1Hum\nKIrSSYwxSE8I8iuKoig9FxUYRVEUJSOowCiKoigZQQVGURRFyQgqMIqiKEpGUIFRFEVRMoIKjKIo\nipIRVGAURVGUjKACoyiKomQEFRhFURQlI6jAKIqiKBlBBUZRFEXJCCowiqIoSkZQgVEURVEyggqM\noiiKkhFUYBRFUZSMoAKjKIqiZAQVGEVRFCUjqMAoiqIoGUEFRlEURckIKjCKoihKRsi5wBhjmowx\nrxhjXjLG1LttfY0xG4wxbxhjao0x5b79rzPGvGmMaTTGTM7dyBVFUZT2yLnAAHuBahEZLyJVbts8\n4EkRORT4HXAdgDFmLHA+UAmcAdxtjDE5GLOiKIrSAfkgMIa24zgbuN89vx/4qnt+FrBWRHaLSBPw\nJlCFoiiKknfkg8AI8IQxZrMx5lK3bYCIfAggIn8DDnDbBwHv+N77ntumKEqe0NLSwubNm2lpacn1\nUJQcU5TrAQAnisgHxpj+wAZjzBtY0fET/zolFixY0Pq8urqa6urqfR2joigpsGbNOmbMmEkwOIyd\nO5tYvvxupky5INfDUtqhrq6Ourq6jBzbiOzT3J0RjDE3ANuAS7FxmQ+NMQOBjSJSaYyZB4iILHb7\nPw7cICLPJTiW5NNnU5SeTktLC0OHjmH79o3AOOBVIpGJbN36J/r375/r4SkpYoxBRNIS286pi8wY\nU2yMKXXPS4DJwGvAo8DFbrevAY+4548CFxpjgsaY4cBIoD6rg1aUXkpHrq+mpiasx3qc2zIOOMht\nV3ojuY7BDAA2GWNeAp4F1ovIBmAxMMm5y04DfgggIg3Ag0AD8BtgppopipJ51qxZx9ChY5g06QqG\nDh3DmjXr2uxTWlrK9u1bgFfdllfZvv0tSktLszpWJX/IKxdZOlEXmaKkh1RdX5s3b+aUUy5g+/ZP\ngKHAVsLhMp566kGOPfbYHI1e6SzpdJHlQ5BfUZQ8pqmpiWBwGNu3R11fgcBQmpqaYgRm2LBhwCfA\nL4ES4H8x5ly3XemN5NpFpihKnjNsmM0I87u+du3a2kY4+vfvz/LldxOJnEtZ2eVEIueyfPndGuDv\nxaiLTFF6KC0tLTQ1NTFs2LAOJ/mO9vXSjwOBoezatbXd9OPOnFfJP9LpIlOBUZQeSHw9yvXXz+Xy\nyy9rVzw6ql1R4egdqMCkgAqM0ltJFJSHEwiHg9x339IY8ehM7YonMDt37mTLli1UVVVRWVmZxU+m\nZIMeUwejKEr68YLysfUoh7Jjx4+ZMWNmTB1Lon29AL4fL035pJO+xkknTeLii69j7NijmT17TqY/\njtKNUYFRlB5GoqA8bAUmtRGPVAL4LS0tzJgxk+3bN7JzZwO2ZO0z4DfcddcyGhsbM/yJlO6KCoyi\n9DCi2VwTgeHASdha5Q/aiId/37KyCUQiE9tkfiW2iIZiU5EHU1+vzTSUxGgdjKL0QKZMuYBPP/2U\n2bO/w65dA4CrCQQKWL783jaxlSlTLuD0009NGsCPtXK8mM5W4H+Bd6mq0hUzlMRokF9ReiDpbjzp\nZZrt2TPAiU0F8DGzZl3GkiV3pHfwSk7RSn5FUdol1ep7j8bGRurr65NmhvmtHM0iU1JFLRhF6YF0\nxoKZPftq7rrrHmAI8I5aJb0crYNJARUYpbeTSvV9Y2MjY8cejc0M8+Irx9PQ8IJaJ70UdZEpitIh\nHQXvAZcBNoTYDDGbGaYCo3QVFRhF6cH079+/3aC+zQB7h9gMMc0MU9KD1sEoSi+msrKSWbMuA44H\nRgPHM2vWZWq9KGlBYzCKonSYRab0HjTInwIqMEpvJx+6Hz/99NNs2LCByZMnc+KJJ+ZkDErn0GaX\niqK0i9ecctKkKxg6dAxr1qzL+hgmTz6Tk06axE03reGkkybxhS+cmfUxKLlFLRhF6WGku4p/X3j6\n6ac56aRJxKc/b9r0hFoyeY5aMIqiJCXVFvyZZMOGDcBgYtOfB7ntSm9BBUZRehiptODPNJMnTwbe\nJXbJgPfcdqW3oAKjKD2MVFrwZ5oTTzyRyZOrsenPo4DjmTy5Wt1jvYy8iMEYYwqA54F3ReQsY0xf\nYB120Ykm4HwR+cTtex1wCbAbmCMiCW1ujcEovR3NIlP2hR6XpmyM+RZwNFDmBGYx8D8icrMx5lqg\nr4jMM8aMBR4AjsU6eJ8ERiVSEhUYpbuQD0KQKt1prMq+0aOC/MaYwcCZwL2+zWcD97vn9wNfdc/P\nAtaKyG4RaQLeBLSnhdJtyYd04lTpTmNV8oOcWzDGmJ8Di4ByYK6zYP4hIn19+3wkIv2MMUuAP4jI\narf9XuA3IvJQguOqBaPkNfmQTpwq3WmsStfoMd2UjTFfAj4UkZeNMdXt7LpPSrFgwYLW59XV1VRX\nt3cKRckuqS4K1hm3VKZcWJ1dwEzpPtTV1VFXV5eZg4tIzh7A94G3gb8AHwDbgBqgERjg9hkINLrn\n84Brfe9/HDguybFFUfKZ5uZmiUT6CbwiIAKvSCTST5qbm1v3Wb16rUQi/aS8fIJEIv1k9eq1SY/X\nmX1TpaGhQVasWCGbNm3qcKxKz8DNnemZ49N1oC4PBE4BHnXPb/aEBLgW+KF7PhZ4CQgCw4EtODdf\nguOl5WIrSibxRKGsbHwbUUhFgPZl31SZNWuOQERgtEBEJk8+I+lYlZ5DOgUmX9eD+SHwoDHmEmAr\ncD6AiDQYYx4EGoBdwEx3QRSlW9LeomCdcUul24XV2NjollGOtnrZsMG2egkGg5pFpqRE3giMiPw3\n8N/u+UfA6Un2+wHwgywOTVEySrJFwWIr8u0kn6wivzP7pkKylS63bNnC1772tX06ptL7yHmasqIo\nielMRX66q/djV7oEXelS2RdynqacKTRNWclH9iXLK1dZZLNnz+Guu5Zha5rfZdasy1iy5I4uHVPJ\nf3pcJX8mUIFR8o1bb72d//zPmwgGh7Nnz9ssX343U6ZckOthtYuudNn7UIFJARUYJZ+49NJvsHz5\nKvxBc3+hok7kSr7Qo1rFKEpPp7GxkeXLVwKj8QfNCwuH0NTUxOzZVzN27NFcfPH3GTv2aGbPnpPD\n0SpK+lCBUZQMYzOyBhEfNN+5s4mdO3f60oHfAJ7lrruW0djYmKPRKkr6UIFRlAxjM68+wNYMTwSO\nBI5n0aL5bNmyhUTpwFaULI2Njdx///0qOkq3QwVGUTJMZWUls2ZdBizA9nR9gxkzpvKd73y7w3Rg\ndZ8p3RkN8itKlmhsbOTJJ59kwIABTJw4sTWNOFk6cGNjI2PHHo0/MQCOp6HhBU0EUDKGBvkVJU9o\naWlh8+bNtLS0dLjfL3/5MNdeu4BvfGNxzHoqS5bcQUPDC6xY8Z80NLzQWmuSrJre7z5TlHxGLRhF\n2UfWrFnHjBkzCQZtm5ZkdS1r1qzjkkuuYMeOncCvgRLgf4lEzm13PRW1YJRcoHUwKaACo2SSVBfg\niu63BPgu8DEwDGgiHC7jqace5Nhjj016Hq2mV7JNj1lwTFG6K6l2L47uNx6bSRa1RnbsOJ7S0tJ2\nz7NkyR3MnHmFFmEq3RIVGEXZB1LtXhzd7yVgJP54SiQygm3btnV4rsrKShWWbsK//vUv3nrrLY44\n4ohcDyUv0CC/ouwDqXYv9vYLh78JvEVsOvL7+9xOX8kf9u7dy8aNG/n617/OoEGDWLJkSa6HlDdo\nDEZRukCq3YtbWlr46U+X8f3v30YgMJRdu7Z2i2aXSnIaGxupqalh1apV9OvXj+nTpzNlyhQOPPDA\nXA+tS2iQPwVUYJR8JJ3t9JXs09LSwtq1a1m5ciXvvfceU6dOZdq0aYwbN67jN3cTsi4wxpggcLCI\nbEnHSbOBCoySLlQUejc7duzgv/7rv1i5ciVPPfUUX/7yl5k+fTqnnXYahYWFuR5e2slqoaUx5kvA\na8AT7vVRxpiH03FyRcl31qxZx9ChY5g06YqY4sjOkGoxppI/iAibNm3i8ssvZ9CgQSxdupRzzz2X\nd955h1WrVjF58uQeKS7ppkMLxhjzAnAasFFExrttr4lIXqdJqAWjdJVUa13aI9ViTCU/2LJlS2tc\nJRwOM336dKZOncrgwYNzPbSske06mF0i8rExMefTmVvp0TQ2NrJ27VoKCg7Cn1qcqNYlGS0tLcyY\nMZPt2ze6eplXmTFjIqeffqq62vKIjz76iAcffJCamhq2bNnClClTePDBB5kwYQJx857SSVIRmEZj\nzPlAgTFmOHAVtlpMUXoks2df7dZoGYLtdHwzcA3Jal2SkWoxppJ9du7cyWOPPcbKlSv57W9/yxe+\n8AWuv/56Jk+eTCAQyPXwegypCMwsYD6wF3gYqAX+M5ODUpRc0djY6FsALNr/q6Skhr17309Y65KM\nVIsxlewgImzevJmVK1eybt06KisrmT59OsuXL6eioiLXw+uRdCgwIvK/2JWSrk33yY0xIeApIOge\nj4jI9caYvsA6YCjQBJwvIp+491wHXALsBuaIyIZ0j0vpvSTrYDx37jnMmjWrU5aHV2Q5Y8bEmNoX\ntV6yy9atW1m1ahUrV65ERJg2bRr19fUMHz4810Pr8aQS5J8AzMN26GsVJBGZkJYBGFMsIv8yxhQC\nTwNzgbOA/xGRm40x1wJ9RWSeMWYs8ABwLLb735PAqETRfA3yK/tCJjoYa5pz9vn000/5xS9+wcqV\nK/njH//IBRdcwLRp0zjuuOM0rtIB2Q7yrwGuw6Yq703HSf2IyL/c0xA2bfofwNnAKW77/UAdVuTO\nAtaKyG6gyRjzJlAFPJfucSk9l/YmfG/1ybvuOh5/B+Ou9ALr37+/CksW2L17N0888QQrV67kscce\n49RTT2XOnDmceeaZhEKhXA+vV5KKBfOMiHwuYwMwpgB4ARgBLBWRa4wx/xCRvr59PhKRfsaYJcAf\nRGS1234v8BsReSjBcdWC6cU0NjYm7ECcatpwsvcr+YWI8PLLL1NTU8Pq1asZPnw406dP5/zzz2e/\n/fbL9fC6Jdm2YG40xvwU6476zNsoIo+mYwAishcYb4wpA2qNMdW0TYPeJ6VYsGBB6/Pq6mqqq6v3\nbZBKtyI+C8xbQ6UzacPawTi/ee+993jggQeoqalh27ZtTJs2jaeeeorRo0fnemjdjrq6Ourq6jJy\n7FQsmJXAEUADUReZiMj0tA/GmO8C24EZQLWIfGiMGYgt8qw0xsxz517s9n8cuEFE2rjI1ILpnbQX\nQ9m2bRuTJl3BJ5+80Lp/WdkEnnzyp+0u+qXkB9u2bePhhx+mpqaG559/nnPPPZfp06dz4oknUlCg\njeHTRbYtmONE5NB0nCweY8z+2ELOT4wxEWAScCPwKHAxsBj4GvCIe8ujwAPGmB8Bg7ALbOgC5Uor\n7a1jf+aZZ2racDdjz549bNy4kZUrV7J+/XpOPPFELr30Uh555BEikUiuh6d0QCoC85wx5lAReSMD\n5z8QuN/YtI4CoEZEfmuMeQl40BhzCbAVOB9ARBqMMQ9iraldwEw1UxQ/VVVV2OLIqIjAu1RVVaU9\nbVizwzLHH//4R2pqanjggQcYOHAg06ZN49Zbb+WAAw7I9dCUTpCKi+w1YDSwBRuDMVg3VVrSlDOF\nush6Lx2tY58OYdAeY+nnww8/ZM2aNaxcuZKWlpbWVviHHXZYrofWq8hqu35jzIhE20XkrXQMIFOo\nwHQ/0mkRZDILLB1NMBXL9u3beeSRR6ipqeGZZ57h7LPPZtq0aVRXV2u34hyRlRiMMabEVfFrj3El\n46TbIshkFpj2GOsae/fu5fe//z01NTU89NBDHHvssUyfPp0HH3yQkpKSXA9PSSNJLRhjzGMicoYx\n5h1smrDx/xSRg7M3zM6jFkz3IVWLIFsxj47OoxbMvvHGG2+0tsIvKytj+vTpXHTRRRx00EG5Hpri\nI50WDCKS8AEMTva77vCwH03pDtTX10t5+QQBaX2UlY2X+vr61n1Wr14rkUg/KS+fIJFIP1m9em2b\n4zQ3N0t9fb00Nzfv81hSOY9/v7Ky8e3u19v5+9//LnfddZccd9xxMnDgQPn2t78tL7/8cq6HlTHS\n8TeYa9zcmZ55OOkv4MV0nSQXDxWY7kNzc7NEIv0EXnEC84pEIv1a/0k7+r1I6sLQlXEk2r+7TyaZ\nYMeOHfLLX/5Szj77bCkvL5cpU6bIY489Jrt27cr10DJKOv4G84FsCcxL6TpJLh4qMN2L9iyCjiyc\nzgpDMlKxpJTE7N27V5555hm54oorZL/99pOJEyfKfffdJ5988kmuh5YV0vU3mA+kU2Daq4MZZIy5\nvR3X2rfT4aJTFIApUy7g9NNPTRj76GhdlXQF3XX9ls7zl7/8hVWrVlFTU0NRURHTp0/nxRdf5OCD\n8zpEm3Y08SMx7QnMduD1bA1EUZJ1He6oQLIzwtBeAF/Xb7F0lOTw8ccfty4x/MYbb3DhhReyevVq\njjnmmF7bCl9vTpKQzLRBYzBKntFezCOVoPvSpfdIKFQmffoc0a6PvDfHVpLFEXbu3CmPPvqonHfe\neVJWVib//u//Lo888oh89tlnOR5x/tBTEj9Io4usvTTlzSLSbTsAappyzyXZHXZ7d94//ekyrrhi\nDrYpxTvAtUQiizW12Efb9OtXCIVOZtq083nkkUcYPXo006dP57zzzqNv374dHa5X0hPaB2Wr2eWl\nxphxyX4pIq+mYwCK0hnaK8iMd7F5/+ylpaXMmXMNsR2WJ1JYeFCv95H7icYRKoAfADXs3LkDYwx/\n+MMfGDEiYVMPxYcuLhdLexbM79t5n4jIyZkZUnpQC6bn0ZkCR78Q7dixBWMOZseO13x7HEko1MQ7\n72zRCQH45z//yYoVK7j66m+zd28JcCHwOcLhq3n77Tf0GvUi0mnBJF1EQUQ+384jr8VF6Zl4d9j+\nVvxepo4f/8Jin3zyAp999gg7dryFtVxwP9/kjjtu7dUT5+7du6mtrWXq1KkMGTKEJ598kquumk04\nXEBZWT2RyLe4776f9OprpHSNDtv1u3Va5gBDReRKY8xIYJSIPJbx0SmKj1QzddqmjFYTDg9A5JRW\n19odd9zB5ZdfltXx5wuvvPJK6xLDQ4YMYdq0adxxxx3sv//+AFx//XXdPo6g5AeprAdzH/Aa8Hn3\n+n3g54AKjJJVUk0jTiRExnzKiy8+w7Zt23rlxPn++++zevVqampq+Pjjj5k2bRq/+93vGDNmTOs+\n/gC1rvCppIWO0syA593Pl3zbXk5XGlumHmiaco+goaFBVqxYIQ0NDa3bUkkj7ikpo11h27ZtsmrV\nKvnCF74gFRUVcskll8jGjRtlz549bfbtKW1OlK5DNtKUPYwxzwCnAs+IyARjzHBgnYhUZVD3uowG\n+bs/s2dfzV133YNdAvkdLrnk/7B8+T0pv78npIx2lr1791JXV0dNTQ2/+tWvOOGEE5g+fTpnnXUW\nxcXFMfv6s+yOPvok7Q6tAFnqpixRS+AMoA5oBu4HmoDT0qVwmXqgFky3pqGhQSAS09sJInLLLbfl\nemh5yeuvvy7z5s2TIUOGyFFHHSW33367fPDBB0n391ssoVCZRCJHaA82RUSybME4ResPfA67Fswz\nItKcFnXLIGrBdB8SWRr3338/F1/8feAN356jCQTe5733/qp31kBzczNr165l5cqVfPDBB61LDB9x\nxBHtvq9tuncdcCb+OiG1YHovWUlT9p3sIaAa2CAiv+oO4qJ0H9asWcfQoWOYNOkKhg4dw5o16wCo\nqqrCVtz7U4vfJRgc0iYtuTexY8cOfv7zn/OVr3yF0aNH8/zzz/ODH/yAt99+m5tvvrlDcYFE6d42\nyy4UOoWysglEIhN7ZQ82Jf2kEoM5DbgA+CLwDLAW+I2I7Mz88PYdtWDyn44KJ2fM+Ab33bcKGAy8\nC/wbkcjjvPDCpl6VDSYibNq0iZqaGn7xi19w9NFHM23aNM455xxKS0s7fbxk1927rqWlpb3q+iqx\nZDUG4z2wKc1nAL8APk6Xjy5TDzQGk5f4M8BSWX/llltuk0CgREpKxkgk0k9mzZrTa7Kd3nzzTZk/\nf74MHz5cDjvsMFm8eLG88847aTl2siy7jrLJOtsItDc3Du2ukI0FxyR2sg4B5wDrsEH+n6RrAJl6\nqMDkH/GT19Kl93S4SFNzc7PU1tZKbW2tNDQ09JhFnZLxP//zP3L33XfLCSecIAcccIBcffXV8uKL\nL8revXvTfq74yb+jRbM6m8qsqc/dk6wKDLAa2ArcC0wCCtN2cuv7+B123ZnXgKvc9r7ABmyEtxYo\n973nOuBNoBGY3M6x03nNlU4SP3k1NDRIKFQmsLF18gqHK2TevOslHK6IuZP23usJkDdBLVy4qEeu\nOPnZZ5/Jww8/LOecc46Ul5fLBRdcIL/+9a9l586dWR1HexZlZ1Zs9G4KwuGKHn0z0FPJtsB8CQik\n64Rxxx4IHOWelzpBGQMsBq5x268FfuiejwVecu66YcAWXBwpwbHTd8WVTuHduZaUHNnq1gqFKgRG\nC/QTWCRwj0Bx6z4LFy6S5ubm1vf26XNEmzTlSKRfj5m09u7dK88++6zMnDlT9t9/fzn55JPl3nvv\nlY8//jhnY2pubpZgsDzm+gaD5Sm7M0Viv3soFljbo24GegO5cJGNcS6yi7xHugYQd55fAacDfwIG\nSFSE/uSezwOu9e3/GHBckmOl63ornSA6SW0UqBd42CcUzQIPCIQTikesC6xe4Mg2E9rChYu6dYX+\nX//6V1m4cKGMHj1aRo8eLQsXLpS//vWvuR6WiNjvLhAoFegrMF6grwQCpdLc3JySBZNoH3us5m59\nM9DbyLYF83+Bp7CFljXu50PpGoDvPMNcfKcU+Efc7z5yP5f4xc257c5Jcrx0Xe9ey74EaGtrawUG\nOktlgkCpwGBnsXjbQgKHxohHJHK4rFixwneX3Oz2bzuhdbfA8ccffyz33nuvnHzyybL//vvLzJkz\n5dlnn81IXKUrRK2UZifwzTFWR0ftdxJZOTBCSkpGd8ubgd5KOgUmlWaXFwBHYZdQnmaMORBYkcL7\nUsYYU4rNTpsjItuMMRK3S/zrlFiwYEHr8+rqaqqrq/d1iL0Obz2VoqJB7Ny5lXnz5jJ8+FCqqqqo\nrKxM+r7nnnsO+ARbtNcIXIH1aM4hWshXhy3sizaj3L79LUaOHOlrUnkg1lg+nj59DmX37rdjajPy\nIX22vVY0u3btYsOGDdTU1PDYY49x+umn861vfYszzzyTYDCYoxG3T7RJ6AfAscR3q54y5QJOP/3U\nmM/c2NhIfX09VVVVCZuMRiL/4KGH1jB+/Pi8+M6UttTV1VFXV5eZg3ekQEC9+/kC0Adbzf+ndCkc\ndvZ5HCsu3rZGYl1kjZLYRfY46iJLO1FXx2JnRRzpXFoDBCIya9ZVSd87depUgZFxFkhbdxccKFDh\nXDH9JBweJvX19bJ69VoJBPo4//1wKSwslnnzrs87a2X16rUSClVIKDRWQqEKWb16rezdu1deeOEF\nufrqq2XAgAFywgknyN133y1///vf9+kcHVlqmbDkOtMkdNasOe7vYnTr34U2Ge3+kGUX2U+BCuCb\n2CD8ZmBl2gYAK4Hb47Yt9oSExEH+IDAcDfJnhPr6ehdkj/enV7jYSiSmu7FHc3Oz3HjjjS7G8oBz\nhyV2d9mJ6WEnPhtj3F+JxO2WW27LG7dYc3OzFBSU+D7PBjEmIGPGjJHhw4fL/Pnz5c9//nOXztFR\nim98IkU6J/JUhCu2V1yDwE0CIWloaOh2LkwllqwJjLNWDvS9HglMSNvJ4URgD/CyE44XsR0D+gFP\nOkHbAFT43nOdExZNU84Qzc3NLqU43uo4ygnCKFmxYkXMe7wJr7h4nJt4As4K8SbhxW774QIROffc\n8xLe6SYWN/vePn3GSyhUIeecc65s2rQpF5dGRESWLVsmMFzgfoHTxAay+8g111zTblwl1Ym3o4B6\ne9le2WLFihXOcom1Yk47bZJs2rRJ5s+fn9PvSNl3sm3B/DFdJ8vmQwWmayxdeo+07WbcL6EFkzh7\nKOx7jHDHOkAgJDNmXBpTQOmfGBsaGlwmkyduiayfYoGwTJ58RsY+fyIx2L17t2zYsEEqKysFEDhZ\n4EEnusUyd+7cpMfrTNFhRynBNpFiZJz4j5Da2tr0XYAOsBZMKMHfSMRtHyUQyeh3pGSGbAvMKmB8\nuk6YrYfKlaDYAAAgAElEQVQKTNdZuvQeCYUqJBAY4yaO/QTCctppk2IEpu2EuNaJwJFi3WoBufvu\nu1sXDks22Xrbw+GDfRNXovjNeLEuuEhG7pLjx7d48S3yH//xH3LQQQfJMcccI3PmzHEWWjSdF4JJ\nx5JqkaK3uNqmTZva3d8KTLHEi242BUZE5LTTTndC4v9uRop1l0UFRy2Z7kVWBAYocj9fB3Y7d9WL\nnisrXQPI1EMFpvMkWz1y4cJFEgiUijEHu7tTKzResD92Am12E27s5HfnnXe2nsMWXW4Uf/ylbRuY\n7wpEpLh4bBJLqllgpMyfPz+t1yD6WX4pcLXASDGmQK666ip5/fXXW/ebPPkMZ50N7tCaqq+vb7Pe\nSjh8mKxYsaJVNOID5pMnn5E0WN5evUo2SbZmj43JRAUn3d+RklmyJTAvup8jEj3SNYBMPVRgOseX\nvvQVgQKx2V2RGPEoLCxxk5lXXT/QWSbBVjHy7vojkVFiXWLie4yQdevWucyrMoGDJBrA7yeh0FCZ\nMWOGlJYeFfe+4VJUFJGCgog73wg3jrUZuTv+17/+Jd/73vfEmLCAcWPsI0VFgxNWoKcaa4idiKPF\nppHIoRKJ9JNbbrkt4US9adOmpDGb1avXSjhcISUloyUcruhykH9fA/OzZl3lxj7K/SxK+DmU7kO2\nBOaldJ0kFw8VmNQZNaoybpI4VLw4iw1ox7tjygVWCBTLokWLWo/jxVUKC0tj9i8q6iMNDQ0uMO1V\n9m9stWDs6/4JJtlygZ9INO15kROakZIu//6ePXtk48aNcskll0hFRYVUVVWJdX/9IW2TZLT41Eu/\nHuk+63UCr0hhYR+BQ+LEtW0iRTzpytbqalNKv+VrLbtIWr8jJbtkS2DeBb6d7JGuAWTqoQKTGqtW\nrUri5thfqqsnSmGhJzz+yW+4m4RL5MILL2wzwdm7674SiRwu4XBfWb16rS8wXSvRGphD3c8BboIN\nCpRJNK5R7EQl7BvfRikqKpH169d36XM3NjbK9ddfLwcffLCMGzdObr31Vnn//fdddlTbuEKiyT7V\nCd5+ds8Ki0/7bhZrmQXbfAeJUsHTTWeaWKaKZpF1b7IlMB8A84EbEj3SNYBMPVRgkuNZGvPmXS/G\nBBJMqCMkmgkUchP9RolNNx4qEJGiokNTWjckGpi+2R2zr9g6mb5ucl3nm3RrBb4h/phEQUG4y8V7\nLS0tsmTJEqmqqpKBAwfK3Llz5eWXX47ZJ1lcIX6y78y6KdbN6N3V+6/zUWLdZcVy1llni9+KbK+Y\nNZ2k2sRS6T1kNQbTXR8qMImJrZQfIbZXWNu7Zyso9wiUOIsjLDbNOCzWbZV66/b6+npf+rEnWPFp\nx7e71+Oc2LSd5NetW9emuWJHFsSOHTvkF7/4hZx11llSXl4uF110kTz++OOya9eupO+JjyvET/b7\nsm5K4rTvYoHy1gB9oiSLTJMJC0bp3mgMRgWmXZK5KJqbm127ey/Lyws4R5xweLGBA524eGIwUqzr\nqtC9PrTN3XhJybjWmhZv0o+faGfNukoKCoKSKAkA5vrE7SBpa1XFxiTasyD27t0rTz/9tFx++eWy\n3377yamnnio/+9nP5NNPP03p+nnJCOHwwRIKlSVs6hifFRaJHN7huile2nc4fJhARILBA9MSoO8q\n2t5F8ZMtgemXrpPk4tGbBMY/qU+a5AVZ2xa61dfXS0nJoc41s1ai3Y2LBb7sBOQrTmz6SNt0Y8+y\nSZyKHAj0kUCgVMrLJ0g4XNGm2jwS6ddOzGekO2aJs2ZC4l+czO+mSjaJP/fcc7JgwQIZMWKEjBkz\nRr7//e/L1q1bO30tO7qjb8+N1p74+L8rb998sRS0vYvikdVCy+766IkCk2gS8N/J2/qStu4uz5Jp\nbm52KbjhJOIRFhjiJvlBEu0l5j1Gis38EokWU8anDnvrf/xEbI1Ic+v7Pd9+fKaRbbdS7/YdKjYO\n4xV32rTpGTMubf3ssXGDjwR+KoWFJdK3b1+56qqrZPPmzfvcCj+VmER9fb0EAgNihLyoqH+rcET7\nrK0QL2sum24vRekKKjC9UGASuYQSt2gpFlgvMF9gk8BI+epXvyoi/vYeXvsWv3iMEOsuaxabHtye\nBeO9Dks4PCxGRGwG2CKx1lGs+Pj7ZW3atEnmzp0rwWCpJF6cTFrfc8MNN8Z89rvuutu9b5JY191k\nCQZL5b333uvyde6cBeONe2OriNhkhqIY8YHCrFfZK8q+ogLTywQm2aRXW1ubYIGnQ8TfCwoKZcKE\nCSIicuedd0o0IyyRMHlCsdYdw4vBjHAT+UUxlseoUWMSCFx5kmP3kcLCtj3MzjvvAid4owVCUlQ0\nJubzlJYe7iyzl91kfqGAkZEjR0ogUCx9+hyR9rhBKgtrWTdYdGEuzw1m64baus+WLVvW7jk1tVfJ\nF1RgepnAJHPb1NbWJpjg462MSKvArFu3zme5eDEYz00V71oLO0EZKjZt2BOfSrHWkbU2brnlttbJ\nOBgsd+m4o+NEb6TYupnBEgzaoLk3iUdjL2slWnTpt2BKJRg8UKzLbITAjVJaOrbVHZWprKv2YhK2\nVYtn4dlUay8T7Prrr5dEdTTXX3990nO1FzdTlGyjAtPLBKa99uzxPbFs/CR2chs6dGjrcYqK+viO\ns9FZFyukrVunRGwsJD6l2OsDJuJldvkn42ivMf97QmItoEMFyqWoqCSBMHrHPVisFTRYoEAikWIp\nLAwJrBTYK7BRQqGyVmHb1+rzdH8fgUCZNDc3y6JFixJaMIsWLWpTG1NfXy/r169PuH+2LRkN8ise\nKjC9UGASNTfctGmTm7wrJNq5OOisi+hk9d3vfrd1Alm69B43Ofoth3onLA1ObNY7sbpZ4GKx8Riv\n5f7imGPHWw+bNm2Sc845V4wpdiLh1b74CysDbt0YvxAeJXCNG38fsa3wD5Bly5b5uiwPF4hIOHx4\nm7FkonYjmYUUtSg3STTWNUIWLlzkvpOiONEvku985xoJhcqkpKRSiopKpKioRMrLJ0hhYVgSWTyZ\naOLZXl+zXIm1kn+owPQygYlOaJ7Pf5OEwwfLhRdemMDCKHYTXLT5YPzd/g033CixacAbJTZofalY\nl5bfoimWCROOidnmL0Bsbm6WqqoT3IR6oNtvmBtP28y2qAXwssA0seurGIEvCmxoHZPXEiaxZRS1\nphJVn3flrjzRcsD+47a9PoVSVFTi3JCeNei5H43vekQEDnM/73GCnhkLxn9TkUxAtNBSiUcFppcJ\nTOwkMMc3eYUk0cJTcKPvznp4G3dOKFQhBQUD3QQ9XqyVUejEYZT72bbqvKioVG644caYu3qvnb89\nxwix7q1yiV/y2E6m3hiHysSJp7qJ17vbnyzWLRfrpvOKKxPFoWzVf33CSbErd+UdtYtJ5tYCZOLE\niXG/84t3oiWom8WYshhB6koMJl5U7OqgIbFrtDS0uVbaKkaJRwWmlwmMSLS6PPHkFT/Recsd28W+\ngsFDJVpnIhIMjpGoBbNIoi4wz2X2gLRNYx4vXtX/0qX3tI7JdgbwW1EPiG2GGT+Zlgn8WOAEAWTq\n1Kny5S9/Wax1s9GJYfI7+cQp2REpLT087Xfl0eWA/Z9/pJx22ukiInLuuedK4oW2CqSkpCTud577\nsV7a1hXZXmSRSD9Zv359l7PIPFHt02e8RF2IF4rfEoOrYgRELRglHhWYXigwIuICyPET2zAnJKNc\nAD9RX7Gw2LVc+km0WaXX3TdREL8hyfZmgXFSVFTiWyDsgbiJs9mJ15ECuwWeFJgudq2ZkEBQiopK\n5JZbbnOfp8CNZUgbUQsGK2PupL0JtLj4CAmFKuSWW25L6ALr6l158oW0QtLQ0CDnnXdekt+XSL9+\n/SR1C6Y4ba1iEgtwX4ntRG3HGQqVJbT2tFWMIqIC02sFxvr3E63u+LCEQmVy++23t5mkowWU/olw\nkJvs50vy5YiDbt/4Sv1+AsNk0aJFvrhQ/MQWEOv6GuCO9x9iraTZYi0Zb9I9xE2ABe4YsZ8tkdvL\npgcPFohIIFCacDJMx135aadNklh33VXiZc1deeWV0jbrrkjAOKvMi7nEx2C82FSlQLFcccWVMRll\nXbEaErsQx4rNyvNvGymXX35lwmumWWSKiApMrxWY5uZmKSgIx01eAyQc7isLFy6Sm266SRIH/f2V\n9iPctuESbV4Zv7/XYflmgdOdCIwTvwVUVBSRwkLvvWvFxl32EzASCoXk2GOrpKio1DdOL17krfEy\nzv3sKzBCAoEyueii/9PuMsGJ7tDD4Yp2M6P29a482vUgGrvw4jA2BlPkfn+A+2lfT5061V27sESF\n3J9A8bBAn1YrIl0ZXMlciIksGG1bo7SHCkwvFRiR2E6/wWCpnHfeBRIOV0hh4aFuQglINJ25zE12\nXgsYT0CsqyoYLJdAYIhARCKRw6WgoMS9v79ELaFmd9w+Aoc78Yk4gQg7kSoTmwV2lMBAue+++6Sh\nocEVHQbde16RaPfmEveetlZGsiaQtlFnW2urpGR0UtdXV+/Kk7Xtr6+vd9cjJLC/eIuvQUgWL14c\nl1TxgCRKxFi4cFHa4x/xorp06T2uU0L215lRui8qML1YYESiE+f69etdfcxG3x2q10dskJv4vHoM\nr3gyKLCfE5NonCAUKmu9O7eLkHkNG732MuUSdWn9TGwqc5lY99bFAn9pvUO+6KJp7v0Hi3ULjZS2\n3ZsPkPigd7I4ibdAmk0oSM2CSReJamGihZYzfcJrrbT58+fHuaqaJd5K9EQkExlciUQ1F+vMKN2X\nHiUwwHLgQ+BV37a+wAbgDaAWKPf97jrgTaARmNzOcdNysfOVaJ3GKDdxr229q7etXbw76z4Su3Lk\nd8UGese0mdhqa2ulvr5evv71GRJNWz5ArDXzrMBZ7hiHiXWfvSuBgJeRNsqJy1SJxlKaJZp63Ffa\num9iYy7hcEXrmjIefheSbUXj9UYrThqDyQa2lqhtoH/9+vVtrJJAoDShu04zuJR8pKcJzEnAUXEC\nsxi4xj2/Fvihez4WeAkoAoYBWwCT5Ljpudp5iI0PhCXW9dVPrCVTLja464lPSGwc5WF3J13rfh+K\nmwT7SDjcV0pKjnRV+J447S82SL2fwOVu28s+Uegr119/vSxatKj1Tjk2xfc2sfGJeDfRSAEkGCyX\nsrLxEgj0kWCwvMNu0ZFIP1m3bl0bIcokiawC27K/MuYzeVlvieI/ydx1msGl5Bs9SmDs52FonMD8\nCRjgng8E/uSezwOu9e33GHBckmOm4VLnB/GT0xVXzHRiMcFnvXiFl152k78o08t0Khe7Tku5FBUd\nJKFQhZSVjZdwuMLXo+wdgavFur6QaKwhJDBICguLJRAokz59jvKlRUer3dum+D7grJe2d/uBwMFS\nW1vr3F+xFk6ybtHZLgJMFoTvyProTPxHM7iUfKI3CMxHcb//yP1cAlzk234vcE6SY3b5QucD3jK7\nffpEA7fxk7GdwPuI7SH2b2JjH8lqOcKtPzdt2iT19fWydu1asXGaSc5a8bLIwk7EPHeWzQibMeNS\nlzKdOEOpb9/+vv29OJDXj2y8RN11gdZOAPEWTrJu0dl0IXUkImp9KD2RdApMEd0D2Zc3LViwoPV5\ndXU11dXVaRpOdvjpT5dxxRVzgGf57LNxwKvMmfN5AoFDgHFur3FAP+AA4Hz38wPg3932vwEHuv0G\nAf8EPgHgrbfeoqGhgfvuuw/4u3v8GjgDeBU4BbjRHetxoBp4leXLj+eQQ4YDB8eNYyA33XQT//jH\nNmA18CjwHnA7sBR4AmjCejc/B2zhueeeY9GiWwDjzmk/565dWxk/fjzLl9/NjBkTCQSGsmvXVpYv\nv5v+/fun5wJ3QFNTE8HgMLZvj37GQGAoTU1N9O/fnylTLuD000+lqamJYcOGxYyrsbGR+vp6qqqq\nqKyszMp4FWVfqKuro66uLjMHT5dSdeVBWwumkVgXWaN7Hu8ie5we5iLzr9luW8Mc5ruzbxZj2hYk\nRi0Tb0GwgyS6WNhIifYGiwh8z20PCxgpKAhIODzKWRrxRZpjxQbp4xMJRrgUZP847PELC/u5/Zud\nJXSgJO4YYLOuZsyY4dxgXpbZeIFiWbhwUZtr0pHlkm5X074G4dtrlJkIXWxMySfogS6yYcBrvteL\nPSEhcZA/CAynhwX5/f7+UKhCgsFBvkl8rUQzshK5mw5wIjHbCYrfjfa822eoWBfY6QK/ctv7im1E\nWZ5EBDb6XnuJBMWybNkymTHjUjeWg33j9BpBPiDR7sHe+PtKtO1/QABf1tVGsQkIP9mn1ONMtZzv\nrBuso0aZ8ehiY0q+0aMEButLeR/4DHgb+Do2TflJbJryBqDCt/91Tlh6VJpyc3NzgtiK19alQqKp\nvl7vr+hyvXbS9oL7Q9x7xjsBmSPRavPJEmsRidjiyLIkInBg3L6jxAv6FxWVuvTbI6SwsETC4cPi\njunFXfxWUbMb2zKBEVJcXCIinb/jT3TtMhmr6YxllLhR5qjWrtB+7NoxuV9sTFH89CiBydSjuwmM\nrbiOdVHZGpOgwLViixyT9f7yt9d/yU3sBW4yv8FnVRRJWyulXGwjzHgRuDnBebyKfq8vWanbf2Ob\niTIQKJGqquMSTKDlrfsvXrw4LeKQTy3nO2PBzJ07VxJ1ZZ47d27Wx60oHiowPUxgonUt8RZMSKxb\nbIREm09uFOvS6iNRF1Sp2MaSpzlROE2iC2J5vcdmSdQd5lkpfdzxk/Uvu0es9TRSCgtLJXa5ZHHH\nWOeeD2hNe/a7km655TbfOCICA8W2pukjIukRh3wrWEzWYiaeZcuWJRSjZcuWZXnEihJFBaaHCUzU\nreIPdPsr3eMXGfOq2YPuNWItllsFtkvUqlkmtu6lTGx843CJutUGu23NYptOFovt8tvPiY63YNhh\nAiH54hfPTCJEd7Y+X7ZsWZuWJPX19W7Rq3qxXQTGiDFlsm7dOhFJnzjkW8pwKu1Z7I1Foe+7jQgU\naksXJaeowPQggWloaJDzz/cWhfI3hCxyItIQ97syga+JjY8c5ib5Fb6JargTnUL3/lKxbin/ypKJ\nAvhhsVbPcImuMhntFGB7b/kbaXrJBaMF+kpBQSSFgsTYwk/vzn716rW+FTH3vQVMdytYrK+vl0hk\nuEQt0FIJh4fpapJKTlGB6SECY4PbYScSnsUwzk3CuO03uUn/NjcxBwSuc8Ijziopk+iKhaUSrdz3\n3GrxVkdYYEzcnfNFUlhYIosWLZLCwojEdwoIhcaKMV7x5RDxOilHIodIKFTWZlnm+ILEYLBUoqto\nemOxsQmb4FARI2i9oSdXVHw3irXwNvaKz63kN+kUmIJ9LJ9RukhjYyN33XUPttfnIcA12A4592Iz\nsIuwiXU3An8FnsJmb5cAFwKV2MLEt4BHgE1AGHga+DPwLFCALa70F0MeBMzBJu71A/4v8ALwAHv2\nDARgzx4D/MFt3whcyWef/YWbb/4BoVCQcNgQCgVZuvQn/Pd/r+XRR39OYWF80eVBNDU1tX5ekQJs\nYeZXsYWX44DB1NfX09TURCh0CHAR0B9/QWNPpn///ixffjeRyLmUlV1OJHJuVgtJFSXTdJdK/h7H\nk08+CQwGJgEzsWJxOFYstgB73J5BrKg8gRWTz4DPYyv23wMqsBX2m7HCFC8m7+GvkLevvwasd8cb\nT1SsPnDvPcjt3+LOVw7s5W9/e5/f/vY3BIPBmMr1xsZGtm/fEnOe7dvforS0lJaWFmbMmMmuXb/3\njeF4rBC+S1VVFfvvvz87dzYRX8k/bNiwrlzibkF73QAUpduTLlMo3x7kuYssdvnj413cI+BiJ1+S\naOZVg0R7g5W7+Ee5299f4NgsiVvifzfO9XaZ73exMZVAoFQaGhpcE0vPZefFborEq6eJz4qKxhK8\nBIV+rbGE+vp6KS09SmJTcccJhOSii6a2HiObQfp0xmq6W9xHUToCjcF0f4FpaGgQY+w67oDACS7u\n4cVSTvLFLBKtPVLi4iD+2E1QbOrxERJNCRaJFmUOE1vjEhG7SNh3BcISCo2QcLgiQWpxvFh5C44V\nxhQDthdLiC7O5T9WP4HhbYoPszFZp7PiP1PdAxQll6jAdHOBmTp1qpSXl8uJJ57kLJaRcUKwnxOL\n4c7K8Bb0Et9jhNhU43gBOcTtf5nYGpb4IsebnVjUuu3DZf78+TFt5i+//Eppu37LOLHpzLaGJhAo\ni5lQ27NAvvOdayS6zHI/8fqWrV+/PqvXPZ31MvlWe6Mo6UIFppsLzO9//3v59NNPRURk1apVcdbC\nue71kWKzw7x04viq+gpp2y6/n9i6F09AvPoWr8hxmBMIr5Aytsp89eq1Lpsr0QqUFW3GED+hJrNA\nbJ3PAe7zHO6OMzBh+5RMks6K/3zqHqAo6UQFppsLTDxHHHGURNew94vNtWI7I9eKrV/xOhVXiC3K\nHCCxVfKegEQk2sIlJHCem9xHSrTexaYne/GUaKrwTWJdbLHdje0x9m1CjbZP2SieCy1R+5RMrx2v\nFoyidIwKTA8TGBGR9evXS3m518JF3ATv1aN4Lfe/KzbG0uCzKu4UG4DvIzZ+Uy7WvdZfoq1mvFqX\nIrFLIJcKFMa4qOyiX8US7SKwWKJFn2FJ1MesMxNqR+1TutrwMlXSmUyQb90DFCUdqMD0QIGJWhB9\n3B1+fMt9f+bXCImuz1Iv0fVXvDYwXsZYoiLLWoFXpKioT0zcJf5u3L53rHgxk2CwXKZPv1jC4b77\nPKEms1BStXDShWaRKUpy0ikwWgeTJ3jFhjt2TAW+iC04PIDYupZB2KLLrdi6mGqgDngXW8NyLLaW\n5F3sOm07495/IHZFhL6Ew9GVGROt3GhrYU7BFnr2JxRaw6xZM7n11pv3uWajsrIy4eqO9fX12BUa\nzsUuDdQEVFBfX5+R1SD79++ftnqTdB5LUXoaWsmfAxobG7n//vtpbGxs3VZaWsqOHX8BTscuUfx3\nrJi86vbwiiQFW4R5JjDK/TwBW7x4JDARu0bbP4CP3PsagYXu/VcDU9m2bQsvvvgyAMOGDfMVOnrn\n+gi7VI99vXv3262icuyxx6Z1Uh05cqQb70ai3QM+dtsVRem2pMsUyrcHeeoii481nHzyKXLooWOk\noCAigcAQiQbtQ84d5s8CK/DFVr7rcycVSzSd2St4vMfFWjxXmdefLCQ2WG8LKz3XTmwMxnO/jRAY\nlPH4Qn19vQSDleJPIAgGKzUjS1FyABqD6Z4C03YxKq9DsZc9NkxgoxgTllCozMVjCt3EXxAjTDbw\nbtdmKS4+QhYtWuTWF/E3lPx2gjhMRKJJAsVSW1srIuIKIsvE33AS+klx8SGt+2QKzchSlPwhnQKj\nLrIsYmMNQ7AxjlOxMZLh2J5f/wZ8CLyHyCDuuus2Fi6cRyAQpqCgCAhhG1i+4X4arCvpVXbtauJ7\n37uN73znJwSDYQoLv0wodBiFhT/B9jvzx1YGA/VEYzKW/v37c+edtwKXYt101tUm8jHjx4/P5GXx\nNX2cSFnZBCKRidr0UVF6AulSqnx7kNcWzMNJLIuDBaoFIrJo0SJpbm6W2tpaufLKKyXR0rqFhX0l\nHK5I2Cq/trY26ZrvngXjzyTzWLr0HgmFyqS09PCsp95qRpai5B7URdY9BUbEqwcJJBAMb62XYoEB\nctVVV7f2uQqHvcW9YoUiECiVdevWtako79PnKFmxYoU0Nze3qT+xLrRxAhUxMRg/OtErSu8lnQKj\nLrIss2TJHaxf/xA2ldiftfUudj2XX1NQsI1ly1ayfftGPvnkBXbsqKOwMAgcB4zGZoxdRiQyioqK\nijYZYP/85xvMnn0bQ4eO4XOf+xwNDS8wf/4UIpGDgXewa878mUhkVMI1VzKRKaYoSu9DBSYHfPnL\nX+a8887CCoUnGDuBAcCXuOyy6QSDw/DHToqLRxIIFALTsKm8M9i1ayvjx49vjV/06TPeHWsB//zn\nq2zfvpEZM2YCsH79b9i+fSvRepkPes2aK4qi5AYVmBzx4x8vIRQKEBWMJ4EPCYWKmDNndhurZPfu\nt1my5EdEIv+PsrKpMYHwKVMu4IUXNvGtb51FSckI7OqY4BVMHnlkFS+99A/s4mUnYutljueb37xU\nrRRFUTJHunxt2XxgS93/hF0b+Nok+3TJD5kN7Fr15a7epFgCgdLWoHqyPleJ4iNeF+Ti4hHStsOy\n14LFe91PbLuY4VJYGNb+WYqixEAaYzDGHq/7YIwpwArLadiF5TcDF4rIn+L2k+7w2VpaWnjppZcA\nGD9+fIxF0dLS0mFblpaWFgYNGsGuXUXYlOctwA6Kiw9lz553KCjoy/btf/G9YwLwHWw68k+IRL7N\n1q1/Snj8VM7f3ufyv/f444/n+eef55hjjuHZZ5+lsbGR+vp6qqqqUm4H05nxdGXsPZFsXo9cXXv9\nztODMQYRMWk5WLqUKlsPbJDhMd/reSSwYugGFkw6qK2tlfguxxCRoqKhEg7bTLG2DS/DAmdIey33\nu7JaY/x7o0WiXiabiSkaTaVzcmfGoytNxpLN65Gra6/fefqgN6cpYzsi3uN7/X+AOxPs16WL3F1Y\nt26dRFv8e49xYtvIvCLBYLmEwxVSVjZegsFyMSYgsKLdivmuVNYn78y8Me714TGv2+uc3JnxaFeA\nWLJ5PXJ17fU7Ty/pFJge3U15wYIFrc+rq6uprq7O2VgyRUVFBTYz7FVsUN9LeR4G9CccPoSf//yH\n9O3bl2HDhvHkk79jxoyZBAJ3sGvX1oQV84m6KwcC0e7L7ZG4M/MgoCTudaPv9eB2Oyd3ZjxdGXtP\nJJvXI1fXXr/zrlFXV0ddXV1mDp4upcrWA+sie9z3ule7yJqbm50brELsSpTeYmHtWyjtFVKqBdNz\nUAtG6Sz0chdZITaSPRSbd/syUJlgvy5e5u7D6tVrJRDoIzBIbMV/RCKRrrV66cpqjfHvjcZcvJU1\nvU84wl0AAAkbSURBVNeJV7fs6nh0pclYsnk9cnXt9TtPH+kUmG6XRQZgjPkicAe2jme5iPwwwT7S\nHT/bvuLPRhsyZAjbtm3rcjaNZpH1HDSLTEmVdGaRdUuBSYXeJjCKoijpIJ0Co5X8iqIoSkZQgVEU\nRVEyggqMoiiKkhFUYBRFUZSMoAKjKIqiZAQVGEVRFCUjqMAoiqIoGUEFRlEURckIKjCKoihKRlCB\nURRFUTKCCoyiKIqSEVRgFEVRlIygAqMoiqJkBBUYRVEUJSOowCiKoigZQQVGURRFyQgqMIqiKEpG\nUIFRFEVRMoIKjKIoipIRVGAURVGUjKACoyiKomQEFRhFURQlI+RMYIwx/26M+aMxZo8xZkLc764z\nxrxpjGk0xkz2bZ9gjHnVGPNnY8z/y/6oFUVRlFTJpQXzGvBvwH/7NxpjKoHzgUrgDOBuY4xxv/4J\nMENERgOjjTFfyOJ4u0xdXV2uh9AGHVPq5OO4dEypoWPKDTkTGBF5Q0TeBEzcr84G1orIbhFpAt4E\nqowxA4E+IrLZ7bcS+GrWBpwG8vEPSseUOvk4Lh1TauiYckM+xmAGAe/4Xr/ntg0C3vVtf9dtUxRF\nUfKQokwe3BjzBDDAvwkQ4D9FZH0mz60oiqLkFiMiuR2AMRuBuSLyons9DxARWexePw7cAGwFNopI\npdt+IXCKiFyZ5Li5/WCKoijdFBGJD13sExm1YDqB/8M8CjxgjPkR1gU2EqgXETHGfGKMqQI2A9OB\nO5MdMF0XSFEURdk3cpmm/FVjzDvA8cB/GWMeAxCRBuBBoAH4DTBTombWN4HlwJ+BN0Xk8eyPXFEU\nRUmFnLvIFEVRlJ5JPmaRdQpjzA3GmHeNMS+6xxd9v8uLgk1jzBeNMX9y57s20+eLO3eTMeYVY8xL\nxph6t62vMWaDMeYNY0ytMabct3/Ca5aGcSw3xnxojHnVt63T40jnd5dkTDn9ezLGDDbG/M4Y87ox\n5jVjzFVue86uVYIxzXbbc3atjDEhY8xz7u/6dWPM9932XF6nZGPK+RxljClw537Uvc7OdRKRbv3A\nJgB8O8H2SuAlbJxpGLCFqMX2HHCse/4b4AsZHF+BO/dQIAC8DIzJ4vX5C9A3btti4Br3/Frgh+75\n2GTXLA3jOAk4Cni1K+NI53eXZEw5/XsCBgJHueelwBvAmFxeq3bGlOtrVex+FgLPAifmwd9UojHl\nfI4CvgWsAh7N5v9et7dgHIkC+vlSsFmFjRdtFZFdwFo3tmxhaGupng3c757fT/Tzn0WCa5aOQYjI\nJuAfXRlHur+7JGOCHP49icjfRORl93wb0AgMJofXKsmYvBq0XF6rf7mnIezf+D/I/d9UojFBDq+T\nMWYwcCZwb9y5M36deorAzDLGvGyMuddn6uVLwWb8OLJdICrAE8aYzcaYS922ASLyIdjJAzjAbU92\nzTLFAZ0cR7a+u7z4ezLGDMNaWM/S+e8sI+Pyjek5tyln18q5fV4C/gbUiU0Qyul1SjImyO3f1I+A\n/8DOBR5ZuU7dQmCMMU8435/3eM39/ApwN3CIiByF/VJvy+1o844TRWQC9g7mm8aYzxP7h0aC17ki\nH8aRF39PxphS4BfAHGc15Pw7SzCmnF4rEdkrIuOxFt7njTHV5Pg6xY3pZGPMKeTwOhljvgR86CzQ\n9ko3MnKd8qUOpl1EZFKKuy4DvA4B7wFDfL8b7LYl254p3gMOzuL5YhCRD9zPFmPMr7Aurw+NMQNE\n5ENn+jb7xprNa9PZcWR8fCLS4nuZk78nY0wRdiKvEZFH3OacXqtEY8qHa+XG8akx5jfAMeTJ35Qb\n06+BY0TE39A329fpROAsY8yZQAToY4ypAf6WlevUlcBRPjyAgb7n3wJWS2ywKggMJzZY9Sx2ojXY\nYNUXMzi+QqJB/iA2yF+ZpWtTDJS65yXA08BkbIDvWkke4GtzzdI0nmHAa77XnR5Hur+7BGPK+d8T\n1r99e9y2nF6rJGPK2bUC9gfK3fMI8BRwWi6vUztjyvnflDvmKUSD/Ddn4zqlZeLI5cP94b+Knbh/\nhfUter+7zl2gRmCyb/vR2OUC3gTuyMIYv4jNvHkTmJfFazPcXZeX3Oed57b3A550Y9oAVHR0zdIw\nltXA+8BnwNvA14G+nR1HOr+7JGPK6d8T9o5zj+97e9H9/XT6O0vXuNoZU86uFXCEG8dLwCvAd/b1\nbzsLY8qLOYpYgcnKddJCS0VRFCUjdIsgv6IoitL9UIFRFEVRMoIKjKIoipIRVGAURVGUjKACoyiK\nomQEFRhFURQlI6jAKEo7GGP2uDbnf3Rt2L/t+93R7bUtN8YMNcZMyc5IFSX/0DoYRWkHY8ynIlLm\nnu8PrAGeFpEFKby3GpgrIl/J6CAVJU9RC0ZRUkRE/g58A5gFYIw5xRiz3vf8JWftvGCMKQF+AJzk\nts1xFs1Txpjn3eN433s3GmN+7hZ5qvHOaYw51hjztOvE+6wxpsR17L3Z2MWtXjbGXJb9q6EoHdMt\nml0qSr4gIn91E3x/b5P7OReYKSJ/MMYUAzuAeVgL5iwAY0wYOF1EdhpjRmKtoWPd+4/C9oH6G/C0\nMeZzwGbs+kHniciLrpvxDmAG8LGIHGeMCbr9N4jI1kx/fkXpDCowitJ5ErU9fxr4kTHmAeAhEXnP\nmDa7BYG7jDFHYXt7jfL9rl5c52tjzMvYJpyfAu+LyIvQutgXbhnbI4wx57n3lrljqcAoeYUKjKJ0\nAmPMIcBuscsftG4XkcXGmP8CvoS1KCYnePu3gL+JyDhjTCGw3fe7z3zP9xD930wkZgaYLSJPdOGj\nKErG0RiMorRP6wTv3GI/AZa02cmYQ0TkdRG5GevaGgP8E2tdeJQDH7jn07FLObTHG8BAY8zR7hyl\nTphqgZlujRaMMaOMMZF9+XCKkknUglGU9gkbY17Eurd2AStF5EcJ9rvaGDMRa328DjyGjc/scUvo\nrgB+DDxkjJkOPA78b5JzCoCI7DLGXIB1q0WAfwGnY9dWHwa8aKwZ1UwX1mxXlEyhacqKoihKRlAX\nmaIoipIRVGAURVGUjKACoyiKomQEFRhFURQlI6jAKIqiKBlBBUZRFEXJCCowiqIoSkZQgVEURVEy\nwv8HAi0rfQXom9sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e139128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Impute missing travel times and re-plot\n",
    "data.impute(column = \"Distance\", by = [\"Origin\", \"Dest\"])\n",
    "scatter_plot(data, \"Distance\", \"TravelTime\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"split\"></a>\n",
    "# Split Data Set into Train and Test Sets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 3 - Fit a model on train; using test as validation.\n",
    "# Create test/train split\n",
    "s = data[\"Year\"].runif()\n",
    "train = data[s <= 0.75]\n",
    "test  = data[s > 0.75]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(337, 32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replace all instances of 'SFO' in the destination column ('Dest') with 'BB8'\n",
    "test[\"Dest\"] = (test[\"Dest\"] == 'SFO').ifelse('BB8', test[\"Dest\"])\n",
    "# print out the number of rows that were effected\n",
    "test[test['Dest']=='BB8'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">*We replace all instances of 'SFO' from the 'Dest' column, to create the situation in which your test set has a categorical level that was not present in the training set (Note: all models will run without breaking, because new categorical levels are interpreted as if they were NA values)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"build\"></a>\n",
    "# Build Models with Supervised Learning Algorithms \n",
    "\n",
    "Train models for GLM, GBM, DRF, Deep Learning & Naive Bayes in one go! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "glm Model Build Progress: [##################################################] 100%\n",
      "\n",
      "gbm Model Build Progress: [##################################################] 100%\n",
      "\n",
      "drf Model Build Progress: [##################################################] 100%\n",
      "\n",
      "deeplearning Model Build Progress: [##################################################] 100%\n",
      "\n",
      "naivebayes Model Build Progress: [##################################################] 100%\n"
     ]
    }
   ],
   "source": [
    "# Set response column\n",
    "myY = \"IsDepDelayed\"\n",
    "# Set feature columns\n",
    "myX = [\"Origin\", \"Dest\", \"Year\", \"UniqueCarrier\", \"DayOfWeek\", \"Month\", \"Distance\", \"FlightNum\"]\n",
    "\n",
    "# Predict delays with GLM\n",
    "data_glm = H2OGeneralizedLinearEstimator(family=\"binomial\", standardize=True)\n",
    "data_glm.train(x = myX, y = myY, training_frame = train, validation_frame = test)\n",
    "\n",
    "# Predict delays with GBM\n",
    "data_gbm2 = H2OGradientBoostingEstimator(balance_classes = False, ntrees = 50, max_depth = 5,\n",
    "                                         distribution = \"bernoulli\", learn_rate = 0.1, min_rows = 2)\n",
    "data_gbm2.train(x = myX, y = myY, training_frame = train, validation_frame = test)\n",
    "\n",
    "# Predict delays with Distributed Random Forest (DRF)\n",
    "data_rf2 = H2ORandomForestEstimator(ntrees = 10,max_depth = 5, balance_classes = False)\n",
    "data_rf2.train(x = myX, y = myY, training_frame = train, validation_frame = test)\n",
    "\n",
    "# Predict delays with Deep Learning\n",
    "data_dl = H2ODeepLearningEstimator(hidden = [10,10], epochs = 5, variable_importances = True,\n",
    "                                   balance_classes = False, loss = \"Automatic\")\n",
    "data_dl.train(x = myX, y = myY, training_frame = train, validation_frame=test)\n",
    "\n",
    "# Predict delays with Naive Bayes\n",
    "# If laplace smoothing is disabled ('laplace=0') the algorithm will predict 0\n",
    "data_nb = H2ONaiveBayesEstimator(laplace=1) \n",
    "data_nb.train(x = myX, y = myY, training_frame = train, validation_frame=test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Or build models individually "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"GLM Model\"></a>\n",
    "`Build GLM` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "glm Model Build Progress: [##################################################] 100%\n",
      "\n",
      "ModelMetricsBinomialGLM: glm\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.21240524065457322\n",
      "R^2: 0.14856313954399736\n",
      "LogLoss: 0.6130051173885511\n",
      "Null degrees of freedom: 11064\n",
      "Residual degrees of freedom: 10830\n",
      "Null deviance: 15316.04643380064\n",
      "Residual deviance: 13565.803247808637\n",
      "AIC: 14035.803247808637\n",
      "AUC: 0.7225305170277084\n",
      "Gini: 0.44506103405541686\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.33560739553025765: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>1542.0</td>\n",
       "<td>3735.0</td>\n",
       "<td>0.7078</td>\n",
       "<td> (3735.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>544.0</td>\n",
       "<td>5244.0</td>\n",
       "<td>0.094</td>\n",
       "<td> (544.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2086.0</td>\n",
       "<td>8979.0</td>\n",
       "<td>0.3867</td>\n",
       "<td> (4279.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     1542  3735   0.7078   (3735.0/5277.0)\n",
       "YES    544   5244   0.094    (544.0/5788.0)\n",
       "Total  2086  8979   0.3867   (4279.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.3356074</td>\n",
       "<td>0.7102323</td>\n",
       "<td>310.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1528930</td>\n",
       "<td>0.8468334</td>\n",
       "<td>384.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5510498</td>\n",
       "<td>0.6883201</td>\n",
       "<td>192.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.4951515</td>\n",
       "<td>0.6703118</td>\n",
       "<td>223.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9865700</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0766047</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9865700</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5510498</td>\n",
       "<td>0.3403036</td>\n",
       "<td>192.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5255959</td>\n",
       "<td>0.6678037</td>\n",
       "<td>206.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.335607     0.710232  310\n",
       "max f2                      0.152893     0.846833  384\n",
       "max f0point5                0.55105      0.68832   192\n",
       "max accuracy                0.495152     0.670312  223\n",
       "max precision               0.98657      1         0\n",
       "max recall                  0.0766047    1         399\n",
       "max specificity             0.98657      1         0\n",
       "max absolute_MCC            0.55105      0.340304  192\n",
       "max min_per_class_accuracy  0.525596     0.667804  206"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.9155318</td>\n",
       "<td>1.8256007</td>\n",
       "<td>1.8256007</td>\n",
       "<td>0.9549550</td>\n",
       "<td>0.9549550</td>\n",
       "<td>0.0183138</td>\n",
       "<td>0.0183138</td>\n",
       "<td>82.5600652</td>\n",
       "<td>82.5600652</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0200633</td>\n",
       "<td>0.8851707</td>\n",
       "<td>1.6878195</td>\n",
       "<td>1.7567101</td>\n",
       "<td>0.8828829</td>\n",
       "<td>0.9189189</td>\n",
       "<td>0.0169316</td>\n",
       "<td>0.0352453</td>\n",
       "<td>68.7819471</td>\n",
       "<td>75.6710062</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.8651742</td>\n",
       "<td>1.6857841</td>\n",
       "<td>1.7332105</td>\n",
       "<td>0.8818182</td>\n",
       "<td>0.9066265</td>\n",
       "<td>0.0167588</td>\n",
       "<td>0.0520041</td>\n",
       "<td>68.5784067</td>\n",
       "<td>73.3210485</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.8503729</td>\n",
       "<td>1.6017062</td>\n",
       "<td>1.7002602</td>\n",
       "<td>0.8378378</td>\n",
       "<td>0.8893905</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0680719</td>\n",
       "<td>60.1706233</td>\n",
       "<td>70.0260210</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.8379788</td>\n",
       "<td>1.5844836</td>\n",
       "<td>1.6770631</td>\n",
       "<td>0.8288288</td>\n",
       "<td>0.8772563</td>\n",
       "<td>0.0158950</td>\n",
       "<td>0.0839668</td>\n",
       "<td>58.4483585</td>\n",
       "<td>67.7063088</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1002259</td>\n",
       "<td>0.7822708</td>\n",
       "<td>1.5913726</td>\n",
       "<td>1.6341792</td>\n",
       "<td>0.8324324</td>\n",
       "<td>0.8548242</td>\n",
       "<td>0.0798203</td>\n",
       "<td>0.1637871</td>\n",
       "<td>59.1372644</td>\n",
       "<td>63.4179232</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.7336886</td>\n",
       "<td>1.3843445</td>\n",
       "<td>1.5512522</td>\n",
       "<td>0.7241379</td>\n",
       "<td>0.8114458</td>\n",
       "<td>0.0689357</td>\n",
       "<td>0.2327229</td>\n",
       "<td>38.4344542</td>\n",
       "<td>55.1252175</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2</td>\n",
       "<td>0.6966957</td>\n",
       "<td>1.3793379</td>\n",
       "<td>1.5082930</td>\n",
       "<td>0.7215190</td>\n",
       "<td>0.7889742</td>\n",
       "<td>0.0689357</td>\n",
       "<td>0.3016586</td>\n",
       "<td>37.9337871</td>\n",
       "<td>50.8293020</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.6390383</td>\n",
       "<td>1.3055607</td>\n",
       "<td>1.4406952</td>\n",
       "<td>0.6829268</td>\n",
       "<td>0.7536145</td>\n",
       "<td>0.1306151</td>\n",
       "<td>0.4322737</td>\n",
       "<td>30.5560706</td>\n",
       "<td>44.0695227</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.5834513</td>\n",
       "<td>1.2237734</td>\n",
       "<td>1.3864893</td>\n",
       "<td>0.6401447</td>\n",
       "<td>0.7252598</td>\n",
       "<td>0.1223220</td>\n",
       "<td>0.5545957</td>\n",
       "<td>22.3773449</td>\n",
       "<td>38.6489288</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5001356</td>\n",
       "<td>0.5292895</td>\n",
       "<td>1.0783585</td>\n",
       "<td>1.3247963</td>\n",
       "<td>0.5640794</td>\n",
       "<td>0.6929888</td>\n",
       "<td>0.1079820</td>\n",
       "<td>0.6625777</td>\n",
       "<td>7.8358467</td>\n",
       "<td>32.4796309</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6</td>\n",
       "<td>0.4600544</td>\n",
       "<td>0.8979000</td>\n",
       "<td>1.2537434</td>\n",
       "<td>0.4696833</td>\n",
       "<td>0.6558217</td>\n",
       "<td>0.0896683</td>\n",
       "<td>0.7522460</td>\n",
       "<td>-10.2099992</td>\n",
       "<td>25.3743377</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.6999548</td>\n",
       "<td>0.4015261</td>\n",
       "<td>0.7639942</td>\n",
       "<td>1.1838063</td>\n",
       "<td>0.3996383</td>\n",
       "<td>0.6192382</td>\n",
       "<td>0.0763649</td>\n",
       "<td>0.8286109</td>\n",
       "<td>-23.6005841</td>\n",
       "<td>18.3806303</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8</td>\n",
       "<td>0.3414800</td>\n",
       "<td>0.6942267</td>\n",
       "<td>1.1225812</td>\n",
       "<td>0.3631436</td>\n",
       "<td>0.5872119</td>\n",
       "<td>0.0694540</td>\n",
       "<td>0.8980650</td>\n",
       "<td>-30.5773275</td>\n",
       "<td>12.2581202</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.8999548</td>\n",
       "<td>0.2668751</td>\n",
       "<td>0.5876878</td>\n",
       "<td>1.0631725</td>\n",
       "<td>0.3074141</td>\n",
       "<td>0.5561358</td>\n",
       "<td>0.0587422</td>\n",
       "<td>0.9568072</td>\n",
       "<td>-41.2312185</td>\n",
       "<td>6.3172477</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0743241</td>\n",
       "<td>0.4317330</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2258356</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0431928</td>\n",
       "<td>1.0</td>\n",
       "<td>-56.8266962</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.915532           1.8256    1.8256             0.954955         0.954955                    0.0183138       0.0183138                  82.5601   82.5601\n",
       "    2        0.0200633                   0.885171           1.68782   1.75671            0.882883         0.918919                    0.0169316       0.0352453                  68.7819   75.671\n",
       "    3        0.0300045                   0.865174           1.68578   1.73321            0.881818         0.906627                    0.0167588       0.0520041                  68.5784   73.321\n",
       "    4        0.0400362                   0.850373           1.60171   1.70026            0.837838         0.889391                    0.0160677       0.0680719                  60.1706   70.026\n",
       "    5        0.0500678                   0.837979           1.58448   1.67706            0.828829         0.877256                    0.015895        0.0839668                  58.4484   67.7063\n",
       "    6        0.100226                    0.782271           1.59137   1.63418            0.832432         0.854824                    0.0798203       0.163787                   59.1373   63.4179\n",
       "    7        0.150023                    0.733689           1.38434   1.55125            0.724138         0.811446                    0.0689357       0.232723                   38.4345   55.1252\n",
       "    8        0.2                         0.696696           1.37934   1.50829            0.721519         0.788974                    0.0689357       0.301659                   37.9338   50.8293\n",
       "    9        0.300045                    0.639038           1.30556   1.4407             0.682927         0.753614                    0.130615        0.432274                   30.5561   44.0695\n",
       "    10       0.4                         0.583451           1.22377   1.38649            0.640145         0.72526                     0.122322        0.554596                   22.3773   38.6489\n",
       "    11       0.500136                    0.52929            1.07836   1.3248             0.564079         0.692989                    0.107982        0.662578                   7.83585   32.4796\n",
       "    12       0.6                         0.460054           0.8979    1.25374            0.469683         0.655822                    0.0896683       0.752246                   -10.21    25.3743\n",
       "    13       0.699955                    0.401526           0.763994  1.18381            0.399638         0.619238                    0.0763649       0.828611                   -23.6006  18.3806\n",
       "    14       0.8                         0.34148            0.694227  1.12258            0.363144         0.587212                    0.069454        0.898065                   -30.5773  12.2581\n",
       "    15       0.899955                    0.266875           0.587688  1.06317            0.307414         0.556136                    0.0587422       0.956807                   -41.2312  6.31725\n",
       "    16       1                           0.0743241          0.431733  1                  0.225836         0.523091                    0.0431928       1                          -56.8267  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set response column\n",
    "myY = \"IsDepDelayed\"\n",
    "# Set feature columns\n",
    "myX = [\"Origin\", \"Dest\", \"Year\", \"UniqueCarrier\", \"DayOfWeek\", \"Month\", \"Distance\", \"FlightNum\"]\n",
    "\n",
    "# Predict delays with GLM\n",
    "data_glm = H2OGeneralizedLinearEstimator(family=\"binomial\", standardize=True)\n",
    "data_glm.train(x = myX, y = myY, training_frame = train, validation_frame = test)\n",
    "\n",
    "data_glm.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "glm prediction Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>predict  </th><th style=\"text-align: right;\">      NO</th><th style=\"text-align: right;\">     YES</th></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.491019</td><td style=\"text-align: right;\">0.508981</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.511203</td><td style=\"text-align: right;\">0.488797</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.418524</td><td style=\"text-align: right;\">0.581476</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.475537</td><td style=\"text-align: right;\">0.524463</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.444683</td><td style=\"text-align: right;\">0.555317</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.381644</td><td style=\"text-align: right;\">0.618356</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.298128</td><td style=\"text-align: right;\">0.701872</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.348571</td><td style=\"text-align: right;\">0.651429</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.356108</td><td style=\"text-align: right;\">0.643892</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.348571</td><td style=\"text-align: right;\">0.651429</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glm_pred_output = data_glm.predict(test)\n",
    "glm_pred_output.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"GBM Model\"></a>\n",
    "`Build GBM`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "gbm Model Build Progress: [##################################################] 100%\n",
      "\n",
      "ModelMetricsBinomial: gbm\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.20758090434261545\n",
      "R^2: 0.1679017290749275\n",
      "LogLoss: 0.602413781737347\n",
      "AUC: 0.7366535927580263\n",
      "Gini: 0.47330718551605266\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3688381072712026: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>2141.0</td>\n",
       "<td>3136.0</td>\n",
       "<td>0.5943</td>\n",
       "<td> (3136.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>800.0</td>\n",
       "<td>4988.0</td>\n",
       "<td>0.1382</td>\n",
       "<td> (800.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2941.0</td>\n",
       "<td>8124.0</td>\n",
       "<td>0.3557</td>\n",
       "<td> (3936.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     2141  3136   0.5943   (3136.0/5277.0)\n",
       "YES    800   4988   0.1382   (800.0/5788.0)\n",
       "Total  2941  8124   0.3557   (3936.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.3688381</td>\n",
       "<td>0.7170788</td>\n",
       "<td>290.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1506781</td>\n",
       "<td>0.8468251</td>\n",
       "<td>382.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5468037</td>\n",
       "<td>0.7011673</td>\n",
       "<td>193.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5217700</td>\n",
       "<td>0.6825124</td>\n",
       "<td>205.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9581786</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0924748</td>\n",
       "<td>1.0</td>\n",
       "<td>396.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9581786</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5217700</td>\n",
       "<td>0.3671079</td>\n",
       "<td>205.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5054973</td>\n",
       "<td>0.6782999</td>\n",
       "<td>215.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.368838     0.717079  290\n",
       "max f2                      0.150678     0.846825  382\n",
       "max f0point5                0.546804     0.701167  193\n",
       "max accuracy                0.52177      0.682512  205\n",
       "max precision               0.958179     1         0\n",
       "max recall                  0.0924748    1         396\n",
       "max specificity             0.958179     1         0\n",
       "max absolute_MCC            0.52177      0.367108  205\n",
       "max min_per_class_accuracy  0.505497     0.6783    215"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.9016550</td>\n",
       "<td>1.8083780</td>\n",
       "<td>1.8083780</td>\n",
       "<td>0.9459459</td>\n",
       "<td>0.9459459</td>\n",
       "<td>0.0181410</td>\n",
       "<td>0.0181410</td>\n",
       "<td>80.8378005</td>\n",
       "<td>80.8378005</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0201536</td>\n",
       "<td>0.8804417</td>\n",
       "<td>1.6898185</td>\n",
       "<td>1.7488324</td>\n",
       "<td>0.8839286</td>\n",
       "<td>0.9147982</td>\n",
       "<td>0.0171044</td>\n",
       "<td>0.0352453</td>\n",
       "<td>68.9818528</td>\n",
       "<td>74.8832438</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.8642542</td>\n",
       "<td>1.6310953</td>\n",
       "<td>1.7101778</td>\n",
       "<td>0.8532110</td>\n",
       "<td>0.8945783</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0513131</td>\n",
       "<td>63.1095338</td>\n",
       "<td>71.0177788</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.8522578</td>\n",
       "<td>1.7222648</td>\n",
       "<td>1.7132064</td>\n",
       "<td>0.9009009</td>\n",
       "<td>0.8961625</td>\n",
       "<td>0.0172771</td>\n",
       "<td>0.0685902</td>\n",
       "<td>72.2264766</td>\n",
       "<td>71.3206354</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.8413600</td>\n",
       "<td>1.7394874</td>\n",
       "<td>1.7184721</td>\n",
       "<td>0.9099099</td>\n",
       "<td>0.8989170</td>\n",
       "<td>0.0174499</td>\n",
       "<td>0.0860401</td>\n",
       "<td>73.9487414</td>\n",
       "<td>71.8472053</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1001356</td>\n",
       "<td>0.7920522</td>\n",
       "<td>1.6011467</td>\n",
       "<td>1.6598094</td>\n",
       "<td>0.8375451</td>\n",
       "<td>0.8682310</td>\n",
       "<td>0.0801659</td>\n",
       "<td>0.1662059</td>\n",
       "<td>60.1146652</td>\n",
       "<td>65.9809353</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1502033</td>\n",
       "<td>0.7486083</td>\n",
       "<td>1.4769198</td>\n",
       "<td>1.5988462</td>\n",
       "<td>0.7725632</td>\n",
       "<td>0.8363418</td>\n",
       "<td>0.0739461</td>\n",
       "<td>0.2401520</td>\n",
       "<td>47.6919757</td>\n",
       "<td>59.8846154</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2002711</td>\n",
       "<td>0.7102250</td>\n",
       "<td>1.4079048</td>\n",
       "<td>1.5511108</td>\n",
       "<td>0.7364621</td>\n",
       "<td>0.8113718</td>\n",
       "<td>0.0704907</td>\n",
       "<td>0.3106427</td>\n",
       "<td>40.7904815</td>\n",
       "<td>55.1110819</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.6530549</td>\n",
       "<td>1.3558623</td>\n",
       "<td>1.4861848</td>\n",
       "<td>0.7092391</td>\n",
       "<td>0.7774096</td>\n",
       "<td>0.1352799</td>\n",
       "<td>0.4459226</td>\n",
       "<td>35.5862298</td>\n",
       "<td>48.6184805</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4009038</td>\n",
       "<td>0.5791790</td>\n",
       "<td>1.1768346</td>\n",
       "<td>1.4083591</td>\n",
       "<td>0.6155914</td>\n",
       "<td>0.7366997</td>\n",
       "<td>0.1186938</td>\n",
       "<td>0.5646164</td>\n",
       "<td>17.6834626</td>\n",
       "<td>40.8359106</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5000452</td>\n",
       "<td>0.5079291</td>\n",
       "<td>1.0926569</td>\n",
       "<td>1.3457664</td>\n",
       "<td>0.5715588</td>\n",
       "<td>0.7039581</td>\n",
       "<td>0.1083276</td>\n",
       "<td>0.6729440</td>\n",
       "<td>9.2656891</td>\n",
       "<td>34.5766421</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6002711</td>\n",
       "<td>0.4436697</td>\n",
       "<td>0.8843185</td>\n",
       "<td>1.2687195</td>\n",
       "<td>0.4625789</td>\n",
       "<td>0.6636555</td>\n",
       "<td>0.0886317</td>\n",
       "<td>0.7615757</td>\n",
       "<td>-11.5681491</td>\n",
       "<td>26.8719487</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.7000452</td>\n",
       "<td>0.3864459</td>\n",
       "<td>0.7497936</td>\n",
       "<td>1.1947595</td>\n",
       "<td>0.3922101</td>\n",
       "<td>0.6249677</td>\n",
       "<td>0.0748100</td>\n",
       "<td>0.8363856</td>\n",
       "<td>-25.0206418</td>\n",
       "<td>19.4759482</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8000904</td>\n",
       "<td>0.3229952</td>\n",
       "<td>0.6475996</td>\n",
       "<td>1.1263413</td>\n",
       "<td>0.3387534</td>\n",
       "<td>0.5891788</td>\n",
       "<td>0.0647892</td>\n",
       "<td>0.9011748</td>\n",
       "<td>-35.2400443</td>\n",
       "<td>12.6341314</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.8999548</td>\n",
       "<td>0.2566938</td>\n",
       "<td>0.5934098</td>\n",
       "<td>1.0672040</td>\n",
       "<td>0.3104072</td>\n",
       "<td>0.5582446</td>\n",
       "<td>0.0592605</td>\n",
       "<td>0.9604354</td>\n",
       "<td>-40.6590168</td>\n",
       "<td>6.7204009</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0676948</td>\n",
       "<td>0.3954675</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2068654</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0395646</td>\n",
       "<td>1.0</td>\n",
       "<td>-60.4532537</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.901655           1.80838   1.80838            0.945946         0.945946                    0.018141        0.018141                   80.8378   80.8378\n",
       "    2        0.0201536                   0.880442           1.68982   1.74883            0.883929         0.914798                    0.0171044       0.0352453                  68.9819   74.8832\n",
       "    3        0.0300045                   0.864254           1.6311    1.71018            0.853211         0.894578                    0.0160677       0.0513131                  63.1095   71.0178\n",
       "    4        0.0400362                   0.852258           1.72226   1.71321            0.900901         0.896163                    0.0172771       0.0685902                  72.2265   71.3206\n",
       "    5        0.0500678                   0.84136            1.73949   1.71847            0.90991          0.898917                    0.0174499       0.0860401                  73.9487   71.8472\n",
       "    6        0.100136                    0.792052           1.60115   1.65981            0.837545         0.868231                    0.0801659       0.166206                   60.1147   65.9809\n",
       "    7        0.150203                    0.748608           1.47692   1.59885            0.772563         0.836342                    0.0739461       0.240152                   47.692    59.8846\n",
       "    8        0.200271                    0.710225           1.4079    1.55111            0.736462         0.811372                    0.0704907       0.310643                   40.7905   55.1111\n",
       "    9        0.300045                    0.653055           1.35586   1.48618            0.709239         0.77741                     0.13528         0.445923                   35.5862   48.6185\n",
       "    10       0.400904                    0.579179           1.17683   1.40836            0.615591         0.7367                      0.118694        0.564616                   17.6835   40.8359\n",
       "    11       0.500045                    0.507929           1.09266   1.34577            0.571559         0.703958                    0.108328        0.672944                   9.26569   34.5766\n",
       "    12       0.600271                    0.44367            0.884319  1.26872            0.462579         0.663656                    0.0886317       0.761576                   -11.5681  26.8719\n",
       "    13       0.700045                    0.386446           0.749794  1.19476            0.39221          0.624968                    0.07481         0.836386                   -25.0206  19.4759\n",
       "    14       0.80009                     0.322995           0.6476    1.12634            0.338753         0.589179                    0.0647892       0.901175                   -35.24    12.6341\n",
       "    15       0.899955                    0.256694           0.59341   1.0672             0.310407         0.558245                    0.0592605       0.960435                   -40.659   6.7204\n",
       "    16       1                           0.0676948          0.395467  1                  0.206865         0.523091                    0.0395646       1                          -60.4533  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set response column\n",
    "myY = \"IsDepDelayed\"\n",
    "# Set feature columns\n",
    "myX = [\"Origin\", \"Dest\", \"Year\", \"UniqueCarrier\", \"DayOfWeek\", \"Month\", \"Distance\", \"FlightNum\"]\n",
    "\n",
    "# Predict delays with GBM\n",
    "data_gbm2 = H2OGradientBoostingEstimator(balance_classes = False, ntrees = 50, max_depth = 5,\n",
    "                                         distribution = \"bernoulli\", learn_rate = 0.1, min_rows = 2)\n",
    "data_gbm2.train(x = myX, y = myY, training_frame = train, validation_frame = test)\n",
    "\n",
    "data_gbm2.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "gbm prediction Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>predict  </th><th style=\"text-align: right;\">      NO</th><th style=\"text-align: right;\">     YES</th></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.726479</td><td style=\"text-align: right;\">0.273521</td></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.748199</td><td style=\"text-align: right;\">0.251801</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.578755</td><td style=\"text-align: right;\">0.421245</td></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.718042</td><td style=\"text-align: right;\">0.281958</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.592616</td><td style=\"text-align: right;\">0.407384</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.30295 </td><td style=\"text-align: right;\">0.69705 </td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.254647</td><td style=\"text-align: right;\">0.745353</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.314306</td><td style=\"text-align: right;\">0.685694</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.308002</td><td style=\"text-align: right;\">0.691998</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.314306</td><td style=\"text-align: right;\">0.685694</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_gbm2.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"DRF Model\"></a>\n",
    "`Build Distributed Random Forest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "drf Model Build Progress: [##################################################] 100%\n",
      "\n",
      "ModelMetricsBinomial: drf\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.21587610782861724\n",
      "R^2: 0.13464999766190133\n",
      "LogLoss: 0.6207972055643648\n",
      "AUC: 0.7172513354494129\n",
      "Gini: 0.43450267089882577\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.40442138612270356: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>1596.0</td>\n",
       "<td>3681.0</td>\n",
       "<td>0.6976</td>\n",
       "<td> (3681.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>579.0</td>\n",
       "<td>5209.0</td>\n",
       "<td>0.1</td>\n",
       "<td> (579.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2175.0</td>\n",
       "<td>8890.0</td>\n",
       "<td>0.385</td>\n",
       "<td> (4260.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     1596  3681   0.6976   (3681.0/5277.0)\n",
       "YES    579   5209   0.1      (579.0/5788.0)\n",
       "Total  2175  8890   0.385    (4260.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.4044214</td>\n",
       "<td>0.7097697</td>\n",
       "<td>316.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.2664590</td>\n",
       "<td>0.8473607</td>\n",
       "<td>390.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5498811</td>\n",
       "<td>0.6798113</td>\n",
       "<td>208.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5140285</td>\n",
       "<td>0.6642567</td>\n",
       "<td>231.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9083639</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.2172136</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9083639</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5140285</td>\n",
       "<td>0.3267948</td>\n",
       "<td>231.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5287009</td>\n",
       "<td>0.6596406</td>\n",
       "<td>222.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.404421     0.70977   316\n",
       "max f2                      0.266459     0.847361  390\n",
       "max f0point5                0.549881     0.679811  208\n",
       "max accuracy                0.514028     0.664257  231\n",
       "max precision               0.908364     1         0\n",
       "max recall                  0.217214     1         399\n",
       "max specificity             0.908364     1         0\n",
       "max absolute_MCC            0.514028     0.326795  231\n",
       "max min_per_class_accuracy  0.528701     0.659641  222"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.8517226</td>\n",
       "<td>1.8600459</td>\n",
       "<td>1.8600459</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.0186593</td>\n",
       "<td>0.0186593</td>\n",
       "<td>86.0045948</td>\n",
       "<td>86.0045948</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0203344</td>\n",
       "<td>0.8352025</td>\n",
       "<td>1.6937114</td>\n",
       "<td>1.7757698</td>\n",
       "<td>0.8859649</td>\n",
       "<td>0.9288889</td>\n",
       "<td>0.0174499</td>\n",
       "<td>0.0361092</td>\n",
       "<td>69.3711430</td>\n",
       "<td>77.5769792</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0301853</td>\n",
       "<td>0.8227487</td>\n",
       "<td>1.7012500</td>\n",
       "<td>1.7514505</td>\n",
       "<td>0.8899083</td>\n",
       "<td>0.9161677</td>\n",
       "<td>0.0167588</td>\n",
       "<td>0.0528680</td>\n",
       "<td>70.1249976</td>\n",
       "<td>75.1450451</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0413918</td>\n",
       "<td>0.8076230</td>\n",
       "<td>1.7267093</td>\n",
       "<td>1.7447520</td>\n",
       "<td>0.9032258</td>\n",
       "<td>0.9126638</td>\n",
       "<td>0.0193504</td>\n",
       "<td>0.0722184</td>\n",
       "<td>72.6709321</td>\n",
       "<td>74.4751979</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0501582</td>\n",
       "<td>0.7957986</td>\n",
       "<td>1.5766713</td>\n",
       "<td>1.7153757</td>\n",
       "<td>0.8247423</td>\n",
       "<td>0.8972973</td>\n",
       "<td>0.0138217</td>\n",
       "<td>0.0860401</td>\n",
       "<td>57.6671250</td>\n",
       "<td>71.5375707</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1000452</td>\n",
       "<td>0.7483208</td>\n",
       "<td>1.5792419</td>\n",
       "<td>1.6474933</td>\n",
       "<td>0.8260870</td>\n",
       "<td>0.8617886</td>\n",
       "<td>0.0787837</td>\n",
       "<td>0.1648238</td>\n",
       "<td>57.9241910</td>\n",
       "<td>64.7493272</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.7075346</td>\n",
       "<td>1.4346497</td>\n",
       "<td>1.5765881</td>\n",
       "<td>0.7504521</td>\n",
       "<td>0.8246988</td>\n",
       "<td>0.0717001</td>\n",
       "<td>0.2365238</td>\n",
       "<td>43.4649665</td>\n",
       "<td>57.6588142</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2004519</td>\n",
       "<td>0.6735833</td>\n",
       "<td>1.3806823</td>\n",
       "<td>1.5273025</td>\n",
       "<td>0.7222222</td>\n",
       "<td>0.7989179</td>\n",
       "<td>0.0696268</td>\n",
       "<td>0.3061507</td>\n",
       "<td>38.0682254</td>\n",
       "<td>52.7302531</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3016719</td>\n",
       "<td>0.6252651</td>\n",
       "<td>1.2733380</td>\n",
       "<td>1.4420897</td>\n",
       "<td>0.6660714</td>\n",
       "<td>0.7543439</td>\n",
       "<td>0.1288874</td>\n",
       "<td>0.4350380</td>\n",
       "<td>27.3338002</td>\n",
       "<td>44.2089747</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.5823499</td>\n",
       "<td>1.1649507</td>\n",
       "<td>1.3739634</td>\n",
       "<td>0.609375</td>\n",
       "<td>0.7187076</td>\n",
       "<td>0.1145473</td>\n",
       "<td>0.5495853</td>\n",
       "<td>16.4950652</td>\n",
       "<td>37.3963372</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5001356</td>\n",
       "<td>0.5308816</td>\n",
       "<td>1.0369495</td>\n",
       "<td>1.3064875</td>\n",
       "<td>0.5424188</td>\n",
       "<td>0.6834116</td>\n",
       "<td>0.1038355</td>\n",
       "<td>0.6534209</td>\n",
       "<td>3.6949502</td>\n",
       "<td>30.6487520</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6000904</td>\n",
       "<td>0.4789212</td>\n",
       "<td>0.9230156</td>\n",
       "<td>1.2426140</td>\n",
       "<td>0.4828210</td>\n",
       "<td>0.65</td>\n",
       "<td>0.0922598</td>\n",
       "<td>0.7456807</td>\n",
       "<td>-7.6984432</td>\n",
       "<td>24.2614029</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.7003163</td>\n",
       "<td>0.4444286</td>\n",
       "<td>0.7757180</td>\n",
       "<td>1.1757941</td>\n",
       "<td>0.4057710</td>\n",
       "<td>0.6150471</td>\n",
       "<td>0.0777471</td>\n",
       "<td>0.8234278</td>\n",
       "<td>-22.4282010</td>\n",
       "<td>17.5794090</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8000904</td>\n",
       "<td>0.4055548</td>\n",
       "<td>0.7463303</td>\n",
       "<td>1.1222385</td>\n",
       "<td>0.3903986</td>\n",
       "<td>0.5870326</td>\n",
       "<td>0.0744644</td>\n",
       "<td>0.8978922</td>\n",
       "<td>-25.3669668</td>\n",
       "<td>12.2238460</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.9001356</td>\n",
       "<td>0.3562070</td>\n",
       "<td>0.5491644</td>\n",
       "<td>1.0585444</td>\n",
       "<td>0.2872629</td>\n",
       "<td>0.5537149</td>\n",
       "<td>0.0549413</td>\n",
       "<td>0.9528334</td>\n",
       "<td>-45.0835576</td>\n",
       "<td>5.8544388</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2172136</td>\n",
       "<td>0.4723058</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2470588</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0471666</td>\n",
       "<td>1.0</td>\n",
       "<td>-52.7694215</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.851723           1.86005   1.86005            0.972973         0.972973                    0.0186593       0.0186593                  86.0046   86.0046\n",
       "    2        0.0203344                   0.835202           1.69371   1.77577            0.885965         0.928889                    0.0174499       0.0361092                  69.3711   77.577\n",
       "    3        0.0301853                   0.822749           1.70125   1.75145            0.889908         0.916168                    0.0167588       0.052868                   70.125    75.145\n",
       "    4        0.0413918                   0.807623           1.72671   1.74475            0.903226         0.912664                    0.0193504       0.0722184                  72.6709   74.4752\n",
       "    5        0.0501582                   0.795799           1.57667   1.71538            0.824742         0.897297                    0.0138217       0.0860401                  57.6671   71.5376\n",
       "    6        0.100045                    0.748321           1.57924   1.64749            0.826087         0.861789                    0.0787837       0.164824                   57.9242   64.7493\n",
       "    7        0.150023                    0.707535           1.43465   1.57659            0.750452         0.824699                    0.0717001       0.236524                   43.465    57.6588\n",
       "    8        0.200452                    0.673583           1.38068   1.5273             0.722222         0.798918                    0.0696268       0.306151                   38.0682   52.7303\n",
       "    9        0.301672                    0.625265           1.27334   1.44209            0.666071         0.754344                    0.128887        0.435038                   27.3338   44.209\n",
       "    10       0.4                         0.58235            1.16495   1.37396            0.609375         0.718708                    0.114547        0.549585                   16.4951   37.3963\n",
       "    11       0.500136                    0.530882           1.03695   1.30649            0.542419         0.683412                    0.103836        0.653421                   3.69495   30.6488\n",
       "    12       0.60009                     0.478921           0.923016  1.24261            0.482821         0.65                        0.0922598       0.745681                   -7.69844  24.2614\n",
       "    13       0.700316                    0.444429           0.775718  1.17579            0.405771         0.615047                    0.0777471       0.823428                   -22.4282  17.5794\n",
       "    14       0.80009                     0.405555           0.74633   1.12224            0.390399         0.587033                    0.0744644       0.897892                   -25.367   12.2238\n",
       "    15       0.900136                    0.356207           0.549164  1.05854            0.287263         0.553715                    0.0549413       0.952833                   -45.0836  5.85444\n",
       "    16       1                           0.217214           0.472306  1                  0.247059         0.523091                    0.0471666       1                          -52.7694  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set response column\n",
    "myY = \"IsDepDelayed\"\n",
    "# Set feature columns\n",
    "myX = [\"Origin\", \"Dest\", \"Year\", \"UniqueCarrier\", \"DayOfWeek\", \"Month\", \"Distance\", \"FlightNum\"]\n",
    "\n",
    "# Predict delays with Distributed Random Forest (DRF)\n",
    "data_rf2 = H2ORandomForestEstimator(ntrees = 10,max_depth = 5, balance_classes = False)\n",
    "data_rf2.train(x = myX, y = myY, training_frame = train, validation_frame = test)\n",
    "\n",
    "data_rf2.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "drf prediction Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>predict  </th><th style=\"text-align: right;\">      NO</th><th style=\"text-align: right;\">     YES</th></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.425925</td><td style=\"text-align: right;\">0.574075</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.437942</td><td style=\"text-align: right;\">0.562058</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.362506</td><td style=\"text-align: right;\">0.637494</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.420964</td><td style=\"text-align: right;\">0.579036</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.348877</td><td style=\"text-align: right;\">0.651123</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.341171</td><td style=\"text-align: right;\">0.658829</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.339797</td><td style=\"text-align: right;\">0.660203</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.341171</td><td style=\"text-align: right;\">0.658829</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.341171</td><td style=\"text-align: right;\">0.658829</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.341171</td><td style=\"text-align: right;\">0.658829</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_rf2.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"DL Model\"></a>\n",
    "`Build Deep Learning`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "deeplearning Model Build Progress: [##################################################] 100%\n",
      "\n",
      "ModelMetricsBinomial: deeplearning\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.2170978543303148\n",
      "R^2: 0.12975256668292567\n",
      "LogLoss: 0.6237081078318155\n",
      "AUC: 0.7209628233723193\n",
      "Gini: 0.44192564674463863\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.29878787453148525: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>2031.0</td>\n",
       "<td>3246.0</td>\n",
       "<td>0.6151</td>\n",
       "<td> (3246.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>789.0</td>\n",
       "<td>4999.0</td>\n",
       "<td>0.1363</td>\n",
       "<td> (789.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2820.0</td>\n",
       "<td>8245.0</td>\n",
       "<td>0.3647</td>\n",
       "<td> (4035.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     2031  3246   0.6151   (3246.0/5277.0)\n",
       "YES    789   4999   0.1363   (789.0/5788.0)\n",
       "Total  2820  8245   0.3647   (4035.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.2987879</td>\n",
       "<td>0.7124635</td>\n",
       "<td>310.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1231821</td>\n",
       "<td>0.8465415</td>\n",
       "<td>392.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.4613361</td>\n",
       "<td>0.6840484</td>\n",
       "<td>220.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.4008738</td>\n",
       "<td>0.6666968</td>\n",
       "<td>254.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9920392</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0999376</td>\n",
       "<td>1.0</td>\n",
       "<td>397.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9920392</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.4613361</td>\n",
       "<td>0.3322544</td>\n",
       "<td>220.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.4315932</td>\n",
       "<td>0.6611711</td>\n",
       "<td>236.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.298788     0.712463  310\n",
       "max f2                      0.123182     0.846541  392\n",
       "max f0point5                0.461336     0.684048  220\n",
       "max accuracy                0.400874     0.666697  254\n",
       "max precision               0.992039     1         0\n",
       "max recall                  0.0999376    1         397\n",
       "max specificity             0.992039     1         0\n",
       "max absolute_MCC            0.461336     0.332254  220\n",
       "max min_per_class_accuracy  0.431593     0.661171  236"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0101220</td>\n",
       "<td>0.9283660</td>\n",
       "<td>1.8093006</td>\n",
       "<td>1.8093006</td>\n",
       "<td>0.9464286</td>\n",
       "<td>0.9464286</td>\n",
       "<td>0.0183138</td>\n",
       "<td>0.0183138</td>\n",
       "<td>80.9300647</td>\n",
       "<td>80.9300647</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0200633</td>\n",
       "<td>0.8904271</td>\n",
       "<td>1.7205425</td>\n",
       "<td>1.7653214</td>\n",
       "<td>0.9</td>\n",
       "<td>0.9234234</td>\n",
       "<td>0.0171044</td>\n",
       "<td>0.0354181</td>\n",
       "<td>72.0542502</td>\n",
       "<td>76.5321386</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.8634937</td>\n",
       "<td>1.6162672</td>\n",
       "<td>1.7159360</td>\n",
       "<td>0.8454545</td>\n",
       "<td>0.8975904</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0514858</td>\n",
       "<td>61.6267199</td>\n",
       "<td>71.5935962</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.8469819</td>\n",
       "<td>1.6705968</td>\n",
       "<td>1.7045756</td>\n",
       "<td>0.8738739</td>\n",
       "<td>0.8916479</td>\n",
       "<td>0.0167588</td>\n",
       "<td>0.0682446</td>\n",
       "<td>67.0596823</td>\n",
       "<td>70.4575591</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.8255683</td>\n",
       "<td>1.6017062</td>\n",
       "<td>1.6839646</td>\n",
       "<td>0.8378378</td>\n",
       "<td>0.8808664</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0843124</td>\n",
       "<td>60.1706233</td>\n",
       "<td>68.3964583</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1000452</td>\n",
       "<td>0.7519715</td>\n",
       "<td>1.5694722</td>\n",
       "<td>1.6267701</td>\n",
       "<td>0.8209765</td>\n",
       "<td>0.8509485</td>\n",
       "<td>0.0784381</td>\n",
       "<td>0.1627505</td>\n",
       "<td>56.9472164</td>\n",
       "<td>62.6770086</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.6936521</td>\n",
       "<td>1.4346497</td>\n",
       "<td>1.5627685</td>\n",
       "<td>0.7504521</td>\n",
       "<td>0.8174699</td>\n",
       "<td>0.0717001</td>\n",
       "<td>0.2344506</td>\n",
       "<td>43.4649665</td>\n",
       "<td>56.2768524</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2</td>\n",
       "<td>0.6494949</td>\n",
       "<td>1.3413110</td>\n",
       "<td>1.5074292</td>\n",
       "<td>0.7016275</td>\n",
       "<td>0.7885224</td>\n",
       "<td>0.0670352</td>\n",
       "<td>0.3014858</td>\n",
       "<td>34.1311012</td>\n",
       "<td>50.7429164</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.5722457</td>\n",
       "<td>1.3245570</td>\n",
       "<td>1.4464534</td>\n",
       "<td>0.6928636</td>\n",
       "<td>0.7566265</td>\n",
       "<td>0.1325155</td>\n",
       "<td>0.4340014</td>\n",
       "<td>32.4556960</td>\n",
       "<td>44.6453402</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.5049965</td>\n",
       "<td>1.1719186</td>\n",
       "<td>1.3778507</td>\n",
       "<td>0.6130199</td>\n",
       "<td>0.7207411</td>\n",
       "<td>0.1171389</td>\n",
       "<td>0.5511403</td>\n",
       "<td>17.1918642</td>\n",
       "<td>37.7850726</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5000452</td>\n",
       "<td>0.4363803</td>\n",
       "<td>1.0447940</td>\n",
       "<td>1.3112153</td>\n",
       "<td>0.5465221</td>\n",
       "<td>0.6858847</td>\n",
       "<td>0.1045266</td>\n",
       "<td>0.6556669</td>\n",
       "<td>4.4793951</td>\n",
       "<td>31.1215293</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6</td>\n",
       "<td>0.3773007</td>\n",
       "<td>0.9558569</td>\n",
       "<td>1.2520157</td>\n",
       "<td>0.5</td>\n",
       "<td>0.6549179</td>\n",
       "<td>0.0955425</td>\n",
       "<td>0.7512094</td>\n",
       "<td>-4.4143055</td>\n",
       "<td>25.2015665</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.6999548</td>\n",
       "<td>0.3224394</td>\n",
       "<td>0.7760936</td>\n",
       "<td>1.1840531</td>\n",
       "<td>0.4059675</td>\n",
       "<td>0.6193673</td>\n",
       "<td>0.0775743</td>\n",
       "<td>0.8287837</td>\n",
       "<td>-22.3906386</td>\n",
       "<td>18.4053135</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8</td>\n",
       "<td>0.2690563</td>\n",
       "<td>0.6907729</td>\n",
       "<td>1.1223652</td>\n",
       "<td>0.3613369</td>\n",
       "<td>0.5870990</td>\n",
       "<td>0.0691085</td>\n",
       "<td>0.8978922</td>\n",
       "<td>-30.9227140</td>\n",
       "<td>12.2365238</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.9000452</td>\n",
       "<td>0.2048145</td>\n",
       "<td>0.5871569</td>\n",
       "<td>1.0628738</td>\n",
       "<td>0.3071364</td>\n",
       "<td>0.5559795</td>\n",
       "<td>0.0587422</td>\n",
       "<td>0.9566344</td>\n",
       "<td>-41.2843069</td>\n",
       "<td>6.2873764</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0899145</td>\n",
       "<td>0.4338519</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2269439</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0433656</td>\n",
       "<td>1.0</td>\n",
       "<td>-56.6148113</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.010122                    0.928366           1.8093    1.8093             0.946429         0.946429                    0.0183138       0.0183138                  80.9301   80.9301\n",
       "    2        0.0200633                   0.890427           1.72054   1.76532            0.9              0.923423                    0.0171044       0.0354181                  72.0543   76.5321\n",
       "    3        0.0300045                   0.863494           1.61627   1.71594            0.845455         0.89759                     0.0160677       0.0514858                  61.6267   71.5936\n",
       "    4        0.0400362                   0.846982           1.6706    1.70458            0.873874         0.891648                    0.0167588       0.0682446                  67.0597   70.4576\n",
       "    5        0.0500678                   0.825568           1.60171   1.68396            0.837838         0.880866                    0.0160677       0.0843124                  60.1706   68.3965\n",
       "    6        0.100045                    0.751971           1.56947   1.62677            0.820976         0.850949                    0.0784381       0.162751                   56.9472   62.677\n",
       "    7        0.150023                    0.693652           1.43465   1.56277            0.750452         0.81747                     0.0717001       0.234451                   43.465    56.2769\n",
       "    8        0.2                         0.649495           1.34131   1.50743            0.701627         0.788522                    0.0670352       0.301486                   34.1311   50.7429\n",
       "    9        0.300045                    0.572246           1.32456   1.44645            0.692864         0.756627                    0.132516        0.434001                   32.4557   44.6453\n",
       "    10       0.4                         0.504997           1.17192   1.37785            0.61302          0.720741                    0.117139        0.55114                    17.1919   37.7851\n",
       "    11       0.500045                    0.43638            1.04479   1.31122            0.546522         0.685885                    0.104527        0.655667                   4.4794    31.1215\n",
       "    12       0.6                         0.377301           0.955857  1.25202            0.5              0.654918                    0.0955425       0.751209                   -4.41431  25.2016\n",
       "    13       0.699955                    0.322439           0.776094  1.18405            0.405967         0.619367                    0.0775743       0.828784                   -22.3906  18.4053\n",
       "    14       0.8                         0.269056           0.690773  1.12237            0.361337         0.587099                    0.0691085       0.897892                   -30.9227  12.2365\n",
       "    15       0.900045                    0.204814           0.587157  1.06287            0.307136         0.55598                     0.0587422       0.956634                   -41.2843  6.28738\n",
       "    16       1                           0.0899145          0.433852  1                  0.226944         0.523091                    0.0433656       1                          -56.6148  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set response column\n",
    "myY = \"IsDepDelayed\"\n",
    "# Set feature columns\n",
    "myX = [\"Origin\", \"Dest\", \"Year\", \"UniqueCarrier\", \"DayOfWeek\", \"Month\", \"Distance\", \"FlightNum\"]\n",
    "\n",
    "# Predict delays with Deep Learning\n",
    "data_dl = H2ODeepLearningEstimator(hidden = [10,10], epochs = 5, variable_importances = True,\n",
    "                                   balance_classes = False, loss = \"Automatic\")\n",
    "data_dl.train(x = myX, y = myY, training_frame = train, validation_frame=test)\n",
    "\n",
    "data_dl.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "deeplearning prediction Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>predict  </th><th style=\"text-align: right;\">      NO</th><th style=\"text-align: right;\">     YES</th></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.772326</td><td style=\"text-align: right;\">0.227674</td></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.856479</td><td style=\"text-align: right;\">0.143521</td></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.702109</td><td style=\"text-align: right;\">0.297891</td></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.761636</td><td style=\"text-align: right;\">0.238364</td></tr>\n",
       "<tr><td>NO       </td><td style=\"text-align: right;\">0.705124</td><td style=\"text-align: right;\">0.294876</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.510008</td><td style=\"text-align: right;\">0.489992</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.285492</td><td style=\"text-align: right;\">0.714508</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.349984</td><td style=\"text-align: right;\">0.650016</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.349114</td><td style=\"text-align: right;\">0.650886</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.349984</td><td style=\"text-align: right;\">0.650016</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dl.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"NB Model\"></a>\n",
    "`Build Naive Bayes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "naivebayes Model Build Progress: [##################################################] 100%\n",
      "\n",
      "ModelMetricsBinomial: naivebayes\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.22952227413516735\n",
      "R^2: 0.07994866844091097\n",
      "LogLoss: 0.6604736519444419\n",
      "AUC: 0.6926305809501246\n",
      "Gini: 0.3852611619002493\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.2227661773519952: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>1169.0</td>\n",
       "<td>4108.0</td>\n",
       "<td>0.7785</td>\n",
       "<td> (4108.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>430.0</td>\n",
       "<td>5358.0</td>\n",
       "<td>0.0743</td>\n",
       "<td> (430.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>1599.0</td>\n",
       "<td>9466.0</td>\n",
       "<td>0.4101</td>\n",
       "<td> (4538.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     1169  4108   0.7785   (4108.0/5277.0)\n",
       "YES    430   5358   0.0743   (430.0/5788.0)\n",
       "Total  1599  9466   0.4101   (4538.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.2227662</td>\n",
       "<td>0.7025043</td>\n",
       "<td>334.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.0499286</td>\n",
       "<td>0.8468321</td>\n",
       "<td>390.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5720444</td>\n",
       "<td>0.6629061</td>\n",
       "<td>187.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5720444</td>\n",
       "<td>0.6419340</td>\n",
       "<td>187.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9972780</td>\n",
       "<td>0.9807692</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0129773</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9972780</td>\n",
       "<td>0.9998105</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5720444</td>\n",
       "<td>0.2928394</td>\n",
       "<td>187.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5013849</td>\n",
       "<td>0.6371044</td>\n",
       "<td>218.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.222766     0.702504  334\n",
       "max f2                      0.0499286    0.846832  390\n",
       "max f0point5                0.572044     0.662906  187\n",
       "max accuracy                0.572044     0.641934  187\n",
       "max precision               0.997278     0.980769  0\n",
       "max recall                  0.0129773    1         399\n",
       "max specificity             0.997278     0.99981   0\n",
       "max absolute_MCC            0.572044     0.292839  187\n",
       "max min_per_class_accuracy  0.501385     0.637104  218"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.9867983</td>\n",
       "<td>1.8600459</td>\n",
       "<td>1.8600459</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.0186593</td>\n",
       "<td>0.0186593</td>\n",
       "<td>86.0045948</td>\n",
       "<td>86.0045948</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0200633</td>\n",
       "<td>0.9728750</td>\n",
       "<td>1.7222648</td>\n",
       "<td>1.7911554</td>\n",
       "<td>0.9009009</td>\n",
       "<td>0.9369369</td>\n",
       "<td>0.0172771</td>\n",
       "<td>0.0359364</td>\n",
       "<td>72.2264766</td>\n",
       "<td>79.1155357</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.9608781</td>\n",
       "<td>1.5815088</td>\n",
       "<td>1.7216941</td>\n",
       "<td>0.8272727</td>\n",
       "<td>0.9006024</td>\n",
       "<td>0.0157222</td>\n",
       "<td>0.0516586</td>\n",
       "<td>58.1508764</td>\n",
       "<td>72.1694137</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.9452227</td>\n",
       "<td>1.4122571</td>\n",
       "<td>1.6441603</td>\n",
       "<td>0.7387387</td>\n",
       "<td>0.8600451</td>\n",
       "<td>0.0141672</td>\n",
       "<td>0.0658258</td>\n",
       "<td>41.2257109</td>\n",
       "<td>64.4160254</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.9312264</td>\n",
       "<td>1.5500383</td>\n",
       "<td>1.6253019</td>\n",
       "<td>0.8108108</td>\n",
       "<td>0.8501805</td>\n",
       "<td>0.0155494</td>\n",
       "<td>0.0813753</td>\n",
       "<td>55.0038290</td>\n",
       "<td>62.5301882</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1000452</td>\n",
       "<td>0.8739324</td>\n",
       "<td>1.5141604</td>\n",
       "<td>1.5697813</td>\n",
       "<td>0.7920434</td>\n",
       "<td>0.8211382</td>\n",
       "<td>0.0756738</td>\n",
       "<td>0.1570491</td>\n",
       "<td>51.4160369</td>\n",
       "<td>56.9781325</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.8246779</td>\n",
       "<td>1.4173647</td>\n",
       "<td>1.5190064</td>\n",
       "<td>0.7414105</td>\n",
       "<td>0.7945783</td>\n",
       "<td>0.0708362</td>\n",
       "<td>0.2278853</td>\n",
       "<td>41.7364729</td>\n",
       "<td>51.9006399</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2</td>\n",
       "<td>0.7696958</td>\n",
       "<td>1.2894562</td>\n",
       "<td>1.4616448</td>\n",
       "<td>0.6745027</td>\n",
       "<td>0.7645730</td>\n",
       "<td>0.0644437</td>\n",
       "<td>0.2923290</td>\n",
       "<td>28.9456205</td>\n",
       "<td>46.1644782</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.6837439</td>\n",
       "<td>1.2882914</td>\n",
       "<td>1.4038429</td>\n",
       "<td>0.6738934</td>\n",
       "<td>0.7343373</td>\n",
       "<td>0.1288874</td>\n",
       "<td>0.4212163</td>\n",
       "<td>28.8291384</td>\n",
       "<td>40.3842911</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.6035968</td>\n",
       "<td>1.1338918</td>\n",
       "<td>1.3363856</td>\n",
       "<td>0.5931284</td>\n",
       "<td>0.6990511</td>\n",
       "<td>0.1133379</td>\n",
       "<td>0.5345543</td>\n",
       "<td>13.3891783</td>\n",
       "<td>33.6385625</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5002259</td>\n",
       "<td>0.5062851</td>\n",
       "<td>0.9705094</td>\n",
       "<td>1.2630782</td>\n",
       "<td>0.5076646</td>\n",
       "<td>0.6607046</td>\n",
       "<td>0.0972702</td>\n",
       "<td>0.6318245</td>\n",
       "<td>-2.9490604</td>\n",
       "<td>26.3078175</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6</td>\n",
       "<td>0.4147244</td>\n",
       "<td>0.8848603</td>\n",
       "<td>1.2001843</td>\n",
       "<td>0.4628623</td>\n",
       "<td>0.6278054</td>\n",
       "<td>0.0882861</td>\n",
       "<td>0.7201106</td>\n",
       "<td>-11.5139676</td>\n",
       "<td>20.0184289</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.6999548</td>\n",
       "<td>0.3256296</td>\n",
       "<td>0.8711608</td>\n",
       "<td>1.1531991</td>\n",
       "<td>0.4556962</td>\n",
       "<td>0.6032279</td>\n",
       "<td>0.0870767</td>\n",
       "<td>0.8071873</td>\n",
       "<td>-12.8839240</td>\n",
       "<td>15.3199135</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8</td>\n",
       "<td>0.2599485</td>\n",
       "<td>0.7650309</td>\n",
       "<td>1.1046562</td>\n",
       "<td>0.4001807</td>\n",
       "<td>0.5778355</td>\n",
       "<td>0.0765377</td>\n",
       "<td>0.8837249</td>\n",
       "<td>-23.4969057</td>\n",
       "<td>10.4656185</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.9000452</td>\n",
       "<td>0.1793016</td>\n",
       "<td>0.6786843</td>\n",
       "<td>1.0573070</td>\n",
       "<td>0.3550136</td>\n",
       "<td>0.5530676</td>\n",
       "<td>0.0678991</td>\n",
       "<td>0.9516240</td>\n",
       "<td>-32.1315665</td>\n",
       "<td>5.7306970</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0098785</td>\n",
       "<td>0.4839782</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2531646</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0483760</td>\n",
       "<td>1.0</td>\n",
       "<td>-51.6021800</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.986798           1.86005   1.86005            0.972973         0.972973                    0.0186593       0.0186593                  86.0046   86.0046\n",
       "    2        0.0200633                   0.972875           1.72226   1.79116            0.900901         0.936937                    0.0172771       0.0359364                  72.2265   79.1155\n",
       "    3        0.0300045                   0.960878           1.58151   1.72169            0.827273         0.900602                    0.0157222       0.0516586                  58.1509   72.1694\n",
       "    4        0.0400362                   0.945223           1.41226   1.64416            0.738739         0.860045                    0.0141672       0.0658258                  41.2257   64.416\n",
       "    5        0.0500678                   0.931226           1.55004   1.6253             0.810811         0.850181                    0.0155494       0.0813753                  55.0038   62.5302\n",
       "    6        0.100045                    0.873932           1.51416   1.56978            0.792043         0.821138                    0.0756738       0.157049                   51.416    56.9781\n",
       "    7        0.150023                    0.824678           1.41736   1.51901            0.74141          0.794578                    0.0708362       0.227885                   41.7365   51.9006\n",
       "    8        0.2                         0.769696           1.28946   1.46164            0.674503         0.764573                    0.0644437       0.292329                   28.9456   46.1645\n",
       "    9        0.300045                    0.683744           1.28829   1.40384            0.673893         0.734337                    0.128887        0.421216                   28.8291   40.3843\n",
       "    10       0.4                         0.603597           1.13389   1.33639            0.593128         0.699051                    0.113338        0.534554                   13.3892   33.6386\n",
       "    11       0.500226                    0.506285           0.970509  1.26308            0.507665         0.660705                    0.0972702       0.631824                   -2.94906  26.3078\n",
       "    12       0.6                         0.414724           0.88486   1.20018            0.462862         0.627805                    0.0882861       0.720111                   -11.514   20.0184\n",
       "    13       0.699955                    0.32563            0.871161  1.1532             0.455696         0.603228                    0.0870767       0.807187                   -12.8839  15.3199\n",
       "    14       0.8                         0.259948           0.765031  1.10466            0.400181         0.577836                    0.0765377       0.883725                   -23.4969  10.4656\n",
       "    15       0.900045                    0.179302           0.678684  1.05731            0.355014         0.553068                    0.0678991       0.951624                   -32.1316  5.7307\n",
       "    16       1                           0.00987846         0.483978  1                  0.253165         0.523091                    0.048376        1                          -51.6022  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set response column\n",
    "myY = \"IsDepDelayed\"\n",
    "# Set feature columns\n",
    "myX = [\"Origin\", \"Dest\", \"Year\", \"UniqueCarrier\", \"DayOfWeek\", \"Month\", \"Distance\", \"FlightNum\"]\n",
    "\n",
    "# Predict delays with Naive Bayes\n",
    "# If laplace smoothing is disabled ('laplace=0') the algorithm will predict 0\n",
    "data_nb = H2ONaiveBayesEstimator(laplace=1) \n",
    "data_nb.train(x = myX, y = myY, training_frame = train, validation_frame=test)\n",
    "\n",
    "data_nb.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "naivebayes prediction Progress: [##################################################] 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>predict  </th><th style=\"text-align: right;\">      NO</th><th style=\"text-align: right;\">     YES</th></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.484349</td><td style=\"text-align: right;\">0.515651</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.479615</td><td style=\"text-align: right;\">0.520385</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.402805</td><td style=\"text-align: right;\">0.597195</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.444515</td><td style=\"text-align: right;\">0.555485</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.353534</td><td style=\"text-align: right;\">0.646466</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.322299</td><td style=\"text-align: right;\">0.677701</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.258183</td><td style=\"text-align: right;\">0.741817</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.292246</td><td style=\"text-align: right;\">0.707754</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.294163</td><td style=\"text-align: right;\">0.705837</td></tr>\n",
       "<tr><td>YES      </td><td style=\"text-align: right;\">0.292246</td><td style=\"text-align: right;\">0.707754</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_nb.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"output\"></a>\n",
    "## Model Performance and Output \n",
    "\n",
    "Run each cell below to see each model's performance on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsBinomialGLM: glm\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.21240524065457322\n",
      "R^2: 0.14856313954399736\n",
      "LogLoss: 0.6130051173885511\n",
      "Null degrees of freedom: 11064\n",
      "Residual degrees of freedom: 10830\n",
      "Null deviance: 15316.04643380064\n",
      "Residual deviance: 13565.803247808637\n",
      "AIC: 14035.803247808637\n",
      "AUC: 0.7225305170277084\n",
      "Gini: 0.44506103405541686\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.33560739553025765: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>1542.0</td>\n",
       "<td>3735.0</td>\n",
       "<td>0.7078</td>\n",
       "<td> (3735.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>544.0</td>\n",
       "<td>5244.0</td>\n",
       "<td>0.094</td>\n",
       "<td> (544.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2086.0</td>\n",
       "<td>8979.0</td>\n",
       "<td>0.3867</td>\n",
       "<td> (4279.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     1542  3735   0.7078   (3735.0/5277.0)\n",
       "YES    544   5244   0.094    (544.0/5788.0)\n",
       "Total  2086  8979   0.3867   (4279.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.3356074</td>\n",
       "<td>0.7102323</td>\n",
       "<td>310.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1528930</td>\n",
       "<td>0.8468334</td>\n",
       "<td>384.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5510498</td>\n",
       "<td>0.6883201</td>\n",
       "<td>192.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.4951515</td>\n",
       "<td>0.6703118</td>\n",
       "<td>223.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9865700</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0766047</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9865700</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5510498</td>\n",
       "<td>0.3403036</td>\n",
       "<td>192.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5255959</td>\n",
       "<td>0.6678037</td>\n",
       "<td>206.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.335607     0.710232  310\n",
       "max f2                      0.152893     0.846833  384\n",
       "max f0point5                0.55105      0.68832   192\n",
       "max accuracy                0.495152     0.670312  223\n",
       "max precision               0.98657      1         0\n",
       "max recall                  0.0766047    1         399\n",
       "max specificity             0.98657      1         0\n",
       "max absolute_MCC            0.55105      0.340304  192\n",
       "max min_per_class_accuracy  0.525596     0.667804  206"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.9155318</td>\n",
       "<td>1.8256007</td>\n",
       "<td>1.8256007</td>\n",
       "<td>0.9549550</td>\n",
       "<td>0.9549550</td>\n",
       "<td>0.0183138</td>\n",
       "<td>0.0183138</td>\n",
       "<td>82.5600652</td>\n",
       "<td>82.5600652</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0200633</td>\n",
       "<td>0.8851707</td>\n",
       "<td>1.6878195</td>\n",
       "<td>1.7567101</td>\n",
       "<td>0.8828829</td>\n",
       "<td>0.9189189</td>\n",
       "<td>0.0169316</td>\n",
       "<td>0.0352453</td>\n",
       "<td>68.7819471</td>\n",
       "<td>75.6710062</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.8651742</td>\n",
       "<td>1.6857841</td>\n",
       "<td>1.7332105</td>\n",
       "<td>0.8818182</td>\n",
       "<td>0.9066265</td>\n",
       "<td>0.0167588</td>\n",
       "<td>0.0520041</td>\n",
       "<td>68.5784067</td>\n",
       "<td>73.3210485</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.8503729</td>\n",
       "<td>1.6017062</td>\n",
       "<td>1.7002602</td>\n",
       "<td>0.8378378</td>\n",
       "<td>0.8893905</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0680719</td>\n",
       "<td>60.1706233</td>\n",
       "<td>70.0260210</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.8379788</td>\n",
       "<td>1.5844836</td>\n",
       "<td>1.6770631</td>\n",
       "<td>0.8288288</td>\n",
       "<td>0.8772563</td>\n",
       "<td>0.0158950</td>\n",
       "<td>0.0839668</td>\n",
       "<td>58.4483585</td>\n",
       "<td>67.7063088</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1002259</td>\n",
       "<td>0.7822708</td>\n",
       "<td>1.5913726</td>\n",
       "<td>1.6341792</td>\n",
       "<td>0.8324324</td>\n",
       "<td>0.8548242</td>\n",
       "<td>0.0798203</td>\n",
       "<td>0.1637871</td>\n",
       "<td>59.1372644</td>\n",
       "<td>63.4179232</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.7336886</td>\n",
       "<td>1.3843445</td>\n",
       "<td>1.5512522</td>\n",
       "<td>0.7241379</td>\n",
       "<td>0.8114458</td>\n",
       "<td>0.0689357</td>\n",
       "<td>0.2327229</td>\n",
       "<td>38.4344542</td>\n",
       "<td>55.1252175</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2</td>\n",
       "<td>0.6966957</td>\n",
       "<td>1.3793379</td>\n",
       "<td>1.5082930</td>\n",
       "<td>0.7215190</td>\n",
       "<td>0.7889742</td>\n",
       "<td>0.0689357</td>\n",
       "<td>0.3016586</td>\n",
       "<td>37.9337871</td>\n",
       "<td>50.8293020</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.6390383</td>\n",
       "<td>1.3055607</td>\n",
       "<td>1.4406952</td>\n",
       "<td>0.6829268</td>\n",
       "<td>0.7536145</td>\n",
       "<td>0.1306151</td>\n",
       "<td>0.4322737</td>\n",
       "<td>30.5560706</td>\n",
       "<td>44.0695227</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.5834513</td>\n",
       "<td>1.2237734</td>\n",
       "<td>1.3864893</td>\n",
       "<td>0.6401447</td>\n",
       "<td>0.7252598</td>\n",
       "<td>0.1223220</td>\n",
       "<td>0.5545957</td>\n",
       "<td>22.3773449</td>\n",
       "<td>38.6489288</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5001356</td>\n",
       "<td>0.5292895</td>\n",
       "<td>1.0783585</td>\n",
       "<td>1.3247963</td>\n",
       "<td>0.5640794</td>\n",
       "<td>0.6929888</td>\n",
       "<td>0.1079820</td>\n",
       "<td>0.6625777</td>\n",
       "<td>7.8358467</td>\n",
       "<td>32.4796309</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6</td>\n",
       "<td>0.4600544</td>\n",
       "<td>0.8979000</td>\n",
       "<td>1.2537434</td>\n",
       "<td>0.4696833</td>\n",
       "<td>0.6558217</td>\n",
       "<td>0.0896683</td>\n",
       "<td>0.7522460</td>\n",
       "<td>-10.2099992</td>\n",
       "<td>25.3743377</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.6999548</td>\n",
       "<td>0.4015261</td>\n",
       "<td>0.7639942</td>\n",
       "<td>1.1838063</td>\n",
       "<td>0.3996383</td>\n",
       "<td>0.6192382</td>\n",
       "<td>0.0763649</td>\n",
       "<td>0.8286109</td>\n",
       "<td>-23.6005841</td>\n",
       "<td>18.3806303</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8</td>\n",
       "<td>0.3414800</td>\n",
       "<td>0.6942267</td>\n",
       "<td>1.1225812</td>\n",
       "<td>0.3631436</td>\n",
       "<td>0.5872119</td>\n",
       "<td>0.0694540</td>\n",
       "<td>0.8980650</td>\n",
       "<td>-30.5773275</td>\n",
       "<td>12.2581202</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.8999548</td>\n",
       "<td>0.2668751</td>\n",
       "<td>0.5876878</td>\n",
       "<td>1.0631725</td>\n",
       "<td>0.3074141</td>\n",
       "<td>0.5561358</td>\n",
       "<td>0.0587422</td>\n",
       "<td>0.9568072</td>\n",
       "<td>-41.2312185</td>\n",
       "<td>6.3172477</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0743241</td>\n",
       "<td>0.4317330</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2258356</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0431928</td>\n",
       "<td>1.0</td>\n",
       "<td>-56.8266962</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.915532           1.8256    1.8256             0.954955         0.954955                    0.0183138       0.0183138                  82.5601   82.5601\n",
       "    2        0.0200633                   0.885171           1.68782   1.75671            0.882883         0.918919                    0.0169316       0.0352453                  68.7819   75.671\n",
       "    3        0.0300045                   0.865174           1.68578   1.73321            0.881818         0.906627                    0.0167588       0.0520041                  68.5784   73.321\n",
       "    4        0.0400362                   0.850373           1.60171   1.70026            0.837838         0.889391                    0.0160677       0.0680719                  60.1706   70.026\n",
       "    5        0.0500678                   0.837979           1.58448   1.67706            0.828829         0.877256                    0.015895        0.0839668                  58.4484   67.7063\n",
       "    6        0.100226                    0.782271           1.59137   1.63418            0.832432         0.854824                    0.0798203       0.163787                   59.1373   63.4179\n",
       "    7        0.150023                    0.733689           1.38434   1.55125            0.724138         0.811446                    0.0689357       0.232723                   38.4345   55.1252\n",
       "    8        0.2                         0.696696           1.37934   1.50829            0.721519         0.788974                    0.0689357       0.301659                   37.9338   50.8293\n",
       "    9        0.300045                    0.639038           1.30556   1.4407             0.682927         0.753614                    0.130615        0.432274                   30.5561   44.0695\n",
       "    10       0.4                         0.583451           1.22377   1.38649            0.640145         0.72526                     0.122322        0.554596                   22.3773   38.6489\n",
       "    11       0.500136                    0.52929            1.07836   1.3248             0.564079         0.692989                    0.107982        0.662578                   7.83585   32.4796\n",
       "    12       0.6                         0.460054           0.8979    1.25374            0.469683         0.655822                    0.0896683       0.752246                   -10.21    25.3743\n",
       "    13       0.699955                    0.401526           0.763994  1.18381            0.399638         0.619238                    0.0763649       0.828611                   -23.6006  18.3806\n",
       "    14       0.8                         0.34148            0.694227  1.12258            0.363144         0.587212                    0.069454        0.898065                   -30.5773  12.2581\n",
       "    15       0.899955                    0.266875           0.587688  1.06317            0.307414         0.556136                    0.0587422       0.956807                   -41.2312  6.31725\n",
       "    16       1                           0.0743241          0.431733  1                  0.225836         0.523091                    0.0431928       1                          -56.8267  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# GLM performance\n",
    "data_glm.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsBinomial: drf\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.21587610782861724\n",
      "R^2: 0.13464999766190133\n",
      "LogLoss: 0.6207972055643648\n",
      "AUC: 0.7172513354494129\n",
      "Gini: 0.43450267089882577\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.40442138612270356: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>1596.0</td>\n",
       "<td>3681.0</td>\n",
       "<td>0.6976</td>\n",
       "<td> (3681.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>579.0</td>\n",
       "<td>5209.0</td>\n",
       "<td>0.1</td>\n",
       "<td> (579.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2175.0</td>\n",
       "<td>8890.0</td>\n",
       "<td>0.385</td>\n",
       "<td> (4260.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     1596  3681   0.6976   (3681.0/5277.0)\n",
       "YES    579   5209   0.1      (579.0/5788.0)\n",
       "Total  2175  8890   0.385    (4260.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.4044214</td>\n",
       "<td>0.7097697</td>\n",
       "<td>316.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.2664590</td>\n",
       "<td>0.8473607</td>\n",
       "<td>390.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5498811</td>\n",
       "<td>0.6798113</td>\n",
       "<td>208.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5140285</td>\n",
       "<td>0.6642567</td>\n",
       "<td>231.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9083639</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.2172136</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9083639</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5140285</td>\n",
       "<td>0.3267948</td>\n",
       "<td>231.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5287009</td>\n",
       "<td>0.6596406</td>\n",
       "<td>222.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.404421     0.70977   316\n",
       "max f2                      0.266459     0.847361  390\n",
       "max f0point5                0.549881     0.679811  208\n",
       "max accuracy                0.514028     0.664257  231\n",
       "max precision               0.908364     1         0\n",
       "max recall                  0.217214     1         399\n",
       "max specificity             0.908364     1         0\n",
       "max absolute_MCC            0.514028     0.326795  231\n",
       "max min_per_class_accuracy  0.528701     0.659641  222"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.8517226</td>\n",
       "<td>1.8600459</td>\n",
       "<td>1.8600459</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.0186593</td>\n",
       "<td>0.0186593</td>\n",
       "<td>86.0045948</td>\n",
       "<td>86.0045948</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0203344</td>\n",
       "<td>0.8352025</td>\n",
       "<td>1.6937114</td>\n",
       "<td>1.7757698</td>\n",
       "<td>0.8859649</td>\n",
       "<td>0.9288889</td>\n",
       "<td>0.0174499</td>\n",
       "<td>0.0361092</td>\n",
       "<td>69.3711430</td>\n",
       "<td>77.5769792</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0301853</td>\n",
       "<td>0.8227487</td>\n",
       "<td>1.7012500</td>\n",
       "<td>1.7514505</td>\n",
       "<td>0.8899083</td>\n",
       "<td>0.9161677</td>\n",
       "<td>0.0167588</td>\n",
       "<td>0.0528680</td>\n",
       "<td>70.1249976</td>\n",
       "<td>75.1450451</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0413918</td>\n",
       "<td>0.8076230</td>\n",
       "<td>1.7267093</td>\n",
       "<td>1.7447520</td>\n",
       "<td>0.9032258</td>\n",
       "<td>0.9126638</td>\n",
       "<td>0.0193504</td>\n",
       "<td>0.0722184</td>\n",
       "<td>72.6709321</td>\n",
       "<td>74.4751979</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0501582</td>\n",
       "<td>0.7957986</td>\n",
       "<td>1.5766713</td>\n",
       "<td>1.7153757</td>\n",
       "<td>0.8247423</td>\n",
       "<td>0.8972973</td>\n",
       "<td>0.0138217</td>\n",
       "<td>0.0860401</td>\n",
       "<td>57.6671250</td>\n",
       "<td>71.5375707</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1000452</td>\n",
       "<td>0.7483208</td>\n",
       "<td>1.5792419</td>\n",
       "<td>1.6474933</td>\n",
       "<td>0.8260870</td>\n",
       "<td>0.8617886</td>\n",
       "<td>0.0787837</td>\n",
       "<td>0.1648238</td>\n",
       "<td>57.9241910</td>\n",
       "<td>64.7493272</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.7075346</td>\n",
       "<td>1.4346497</td>\n",
       "<td>1.5765881</td>\n",
       "<td>0.7504521</td>\n",
       "<td>0.8246988</td>\n",
       "<td>0.0717001</td>\n",
       "<td>0.2365238</td>\n",
       "<td>43.4649665</td>\n",
       "<td>57.6588142</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2004519</td>\n",
       "<td>0.6735833</td>\n",
       "<td>1.3806823</td>\n",
       "<td>1.5273025</td>\n",
       "<td>0.7222222</td>\n",
       "<td>0.7989179</td>\n",
       "<td>0.0696268</td>\n",
       "<td>0.3061507</td>\n",
       "<td>38.0682254</td>\n",
       "<td>52.7302531</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3016719</td>\n",
       "<td>0.6252651</td>\n",
       "<td>1.2733380</td>\n",
       "<td>1.4420897</td>\n",
       "<td>0.6660714</td>\n",
       "<td>0.7543439</td>\n",
       "<td>0.1288874</td>\n",
       "<td>0.4350380</td>\n",
       "<td>27.3338002</td>\n",
       "<td>44.2089747</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.5823499</td>\n",
       "<td>1.1649507</td>\n",
       "<td>1.3739634</td>\n",
       "<td>0.609375</td>\n",
       "<td>0.7187076</td>\n",
       "<td>0.1145473</td>\n",
       "<td>0.5495853</td>\n",
       "<td>16.4950652</td>\n",
       "<td>37.3963372</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5001356</td>\n",
       "<td>0.5308816</td>\n",
       "<td>1.0369495</td>\n",
       "<td>1.3064875</td>\n",
       "<td>0.5424188</td>\n",
       "<td>0.6834116</td>\n",
       "<td>0.1038355</td>\n",
       "<td>0.6534209</td>\n",
       "<td>3.6949502</td>\n",
       "<td>30.6487520</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6000904</td>\n",
       "<td>0.4789212</td>\n",
       "<td>0.9230156</td>\n",
       "<td>1.2426140</td>\n",
       "<td>0.4828210</td>\n",
       "<td>0.65</td>\n",
       "<td>0.0922598</td>\n",
       "<td>0.7456807</td>\n",
       "<td>-7.6984432</td>\n",
       "<td>24.2614029</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.7003163</td>\n",
       "<td>0.4444286</td>\n",
       "<td>0.7757180</td>\n",
       "<td>1.1757941</td>\n",
       "<td>0.4057710</td>\n",
       "<td>0.6150471</td>\n",
       "<td>0.0777471</td>\n",
       "<td>0.8234278</td>\n",
       "<td>-22.4282010</td>\n",
       "<td>17.5794090</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8000904</td>\n",
       "<td>0.4055548</td>\n",
       "<td>0.7463303</td>\n",
       "<td>1.1222385</td>\n",
       "<td>0.3903986</td>\n",
       "<td>0.5870326</td>\n",
       "<td>0.0744644</td>\n",
       "<td>0.8978922</td>\n",
       "<td>-25.3669668</td>\n",
       "<td>12.2238460</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.9001356</td>\n",
       "<td>0.3562070</td>\n",
       "<td>0.5491644</td>\n",
       "<td>1.0585444</td>\n",
       "<td>0.2872629</td>\n",
       "<td>0.5537149</td>\n",
       "<td>0.0549413</td>\n",
       "<td>0.9528334</td>\n",
       "<td>-45.0835576</td>\n",
       "<td>5.8544388</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2172136</td>\n",
       "<td>0.4723058</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2470588</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0471666</td>\n",
       "<td>1.0</td>\n",
       "<td>-52.7694215</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.851723           1.86005   1.86005            0.972973         0.972973                    0.0186593       0.0186593                  86.0046   86.0046\n",
       "    2        0.0203344                   0.835202           1.69371   1.77577            0.885965         0.928889                    0.0174499       0.0361092                  69.3711   77.577\n",
       "    3        0.0301853                   0.822749           1.70125   1.75145            0.889908         0.916168                    0.0167588       0.052868                   70.125    75.145\n",
       "    4        0.0413918                   0.807623           1.72671   1.74475            0.903226         0.912664                    0.0193504       0.0722184                  72.6709   74.4752\n",
       "    5        0.0501582                   0.795799           1.57667   1.71538            0.824742         0.897297                    0.0138217       0.0860401                  57.6671   71.5376\n",
       "    6        0.100045                    0.748321           1.57924   1.64749            0.826087         0.861789                    0.0787837       0.164824                   57.9242   64.7493\n",
       "    7        0.150023                    0.707535           1.43465   1.57659            0.750452         0.824699                    0.0717001       0.236524                   43.465    57.6588\n",
       "    8        0.200452                    0.673583           1.38068   1.5273             0.722222         0.798918                    0.0696268       0.306151                   38.0682   52.7303\n",
       "    9        0.301672                    0.625265           1.27334   1.44209            0.666071         0.754344                    0.128887        0.435038                   27.3338   44.209\n",
       "    10       0.4                         0.58235            1.16495   1.37396            0.609375         0.718708                    0.114547        0.549585                   16.4951   37.3963\n",
       "    11       0.500136                    0.530882           1.03695   1.30649            0.542419         0.683412                    0.103836        0.653421                   3.69495   30.6488\n",
       "    12       0.60009                     0.478921           0.923016  1.24261            0.482821         0.65                        0.0922598       0.745681                   -7.69844  24.2614\n",
       "    13       0.700316                    0.444429           0.775718  1.17579            0.405771         0.615047                    0.0777471       0.823428                   -22.4282  17.5794\n",
       "    14       0.80009                     0.405555           0.74633   1.12224            0.390399         0.587033                    0.0744644       0.897892                   -25.367   12.2238\n",
       "    15       0.900136                    0.356207           0.549164  1.05854            0.287263         0.553715                    0.0549413       0.952833                   -45.0836  5.85444\n",
       "    16       1                           0.217214           0.472306  1                  0.247059         0.523091                    0.0471666       1                          -52.7694  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Distributed Random Forest Performance\n",
    "data_rf2.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsBinomial: gbm\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.20758090434261545\n",
      "R^2: 0.1679017290749275\n",
      "LogLoss: 0.602413781737347\n",
      "AUC: 0.7366535927580263\n",
      "Gini: 0.47330718551605266\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3688381072712026: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>2141.0</td>\n",
       "<td>3136.0</td>\n",
       "<td>0.5943</td>\n",
       "<td> (3136.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>800.0</td>\n",
       "<td>4988.0</td>\n",
       "<td>0.1382</td>\n",
       "<td> (800.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2941.0</td>\n",
       "<td>8124.0</td>\n",
       "<td>0.3557</td>\n",
       "<td> (3936.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     2141  3136   0.5943   (3136.0/5277.0)\n",
       "YES    800   4988   0.1382   (800.0/5788.0)\n",
       "Total  2941  8124   0.3557   (3936.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.3688381</td>\n",
       "<td>0.7170788</td>\n",
       "<td>290.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1506781</td>\n",
       "<td>0.8468251</td>\n",
       "<td>382.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5468037</td>\n",
       "<td>0.7011673</td>\n",
       "<td>193.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5217700</td>\n",
       "<td>0.6825124</td>\n",
       "<td>205.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9581786</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0924748</td>\n",
       "<td>1.0</td>\n",
       "<td>396.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9581786</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5217700</td>\n",
       "<td>0.3671079</td>\n",
       "<td>205.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5054973</td>\n",
       "<td>0.6782999</td>\n",
       "<td>215.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.368838     0.717079  290\n",
       "max f2                      0.150678     0.846825  382\n",
       "max f0point5                0.546804     0.701167  193\n",
       "max accuracy                0.52177      0.682512  205\n",
       "max precision               0.958179     1         0\n",
       "max recall                  0.0924748    1         396\n",
       "max specificity             0.958179     1         0\n",
       "max absolute_MCC            0.52177      0.367108  205\n",
       "max min_per_class_accuracy  0.505497     0.6783    215"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.9016550</td>\n",
       "<td>1.8083780</td>\n",
       "<td>1.8083780</td>\n",
       "<td>0.9459459</td>\n",
       "<td>0.9459459</td>\n",
       "<td>0.0181410</td>\n",
       "<td>0.0181410</td>\n",
       "<td>80.8378005</td>\n",
       "<td>80.8378005</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0201536</td>\n",
       "<td>0.8804417</td>\n",
       "<td>1.6898185</td>\n",
       "<td>1.7488324</td>\n",
       "<td>0.8839286</td>\n",
       "<td>0.9147982</td>\n",
       "<td>0.0171044</td>\n",
       "<td>0.0352453</td>\n",
       "<td>68.9818528</td>\n",
       "<td>74.8832438</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.8642542</td>\n",
       "<td>1.6310953</td>\n",
       "<td>1.7101778</td>\n",
       "<td>0.8532110</td>\n",
       "<td>0.8945783</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0513131</td>\n",
       "<td>63.1095338</td>\n",
       "<td>71.0177788</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.8522578</td>\n",
       "<td>1.7222648</td>\n",
       "<td>1.7132064</td>\n",
       "<td>0.9009009</td>\n",
       "<td>0.8961625</td>\n",
       "<td>0.0172771</td>\n",
       "<td>0.0685902</td>\n",
       "<td>72.2264766</td>\n",
       "<td>71.3206354</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.8413600</td>\n",
       "<td>1.7394874</td>\n",
       "<td>1.7184721</td>\n",
       "<td>0.9099099</td>\n",
       "<td>0.8989170</td>\n",
       "<td>0.0174499</td>\n",
       "<td>0.0860401</td>\n",
       "<td>73.9487414</td>\n",
       "<td>71.8472053</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1001356</td>\n",
       "<td>0.7920522</td>\n",
       "<td>1.6011467</td>\n",
       "<td>1.6598094</td>\n",
       "<td>0.8375451</td>\n",
       "<td>0.8682310</td>\n",
       "<td>0.0801659</td>\n",
       "<td>0.1662059</td>\n",
       "<td>60.1146652</td>\n",
       "<td>65.9809353</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1502033</td>\n",
       "<td>0.7486083</td>\n",
       "<td>1.4769198</td>\n",
       "<td>1.5988462</td>\n",
       "<td>0.7725632</td>\n",
       "<td>0.8363418</td>\n",
       "<td>0.0739461</td>\n",
       "<td>0.2401520</td>\n",
       "<td>47.6919757</td>\n",
       "<td>59.8846154</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2002711</td>\n",
       "<td>0.7102250</td>\n",
       "<td>1.4079048</td>\n",
       "<td>1.5511108</td>\n",
       "<td>0.7364621</td>\n",
       "<td>0.8113718</td>\n",
       "<td>0.0704907</td>\n",
       "<td>0.3106427</td>\n",
       "<td>40.7904815</td>\n",
       "<td>55.1110819</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.6530549</td>\n",
       "<td>1.3558623</td>\n",
       "<td>1.4861848</td>\n",
       "<td>0.7092391</td>\n",
       "<td>0.7774096</td>\n",
       "<td>0.1352799</td>\n",
       "<td>0.4459226</td>\n",
       "<td>35.5862298</td>\n",
       "<td>48.6184805</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4009038</td>\n",
       "<td>0.5791790</td>\n",
       "<td>1.1768346</td>\n",
       "<td>1.4083591</td>\n",
       "<td>0.6155914</td>\n",
       "<td>0.7366997</td>\n",
       "<td>0.1186938</td>\n",
       "<td>0.5646164</td>\n",
       "<td>17.6834626</td>\n",
       "<td>40.8359106</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5000452</td>\n",
       "<td>0.5079291</td>\n",
       "<td>1.0926569</td>\n",
       "<td>1.3457664</td>\n",
       "<td>0.5715588</td>\n",
       "<td>0.7039581</td>\n",
       "<td>0.1083276</td>\n",
       "<td>0.6729440</td>\n",
       "<td>9.2656891</td>\n",
       "<td>34.5766421</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6002711</td>\n",
       "<td>0.4436697</td>\n",
       "<td>0.8843185</td>\n",
       "<td>1.2687195</td>\n",
       "<td>0.4625789</td>\n",
       "<td>0.6636555</td>\n",
       "<td>0.0886317</td>\n",
       "<td>0.7615757</td>\n",
       "<td>-11.5681491</td>\n",
       "<td>26.8719487</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.7000452</td>\n",
       "<td>0.3864459</td>\n",
       "<td>0.7497936</td>\n",
       "<td>1.1947595</td>\n",
       "<td>0.3922101</td>\n",
       "<td>0.6249677</td>\n",
       "<td>0.0748100</td>\n",
       "<td>0.8363856</td>\n",
       "<td>-25.0206418</td>\n",
       "<td>19.4759482</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8000904</td>\n",
       "<td>0.3229952</td>\n",
       "<td>0.6475996</td>\n",
       "<td>1.1263413</td>\n",
       "<td>0.3387534</td>\n",
       "<td>0.5891788</td>\n",
       "<td>0.0647892</td>\n",
       "<td>0.9011748</td>\n",
       "<td>-35.2400443</td>\n",
       "<td>12.6341314</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.8999548</td>\n",
       "<td>0.2566938</td>\n",
       "<td>0.5934098</td>\n",
       "<td>1.0672040</td>\n",
       "<td>0.3104072</td>\n",
       "<td>0.5582446</td>\n",
       "<td>0.0592605</td>\n",
       "<td>0.9604354</td>\n",
       "<td>-40.6590168</td>\n",
       "<td>6.7204009</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0676948</td>\n",
       "<td>0.3954675</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2068654</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0395646</td>\n",
       "<td>1.0</td>\n",
       "<td>-60.4532537</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.901655           1.80838   1.80838            0.945946         0.945946                    0.018141        0.018141                   80.8378   80.8378\n",
       "    2        0.0201536                   0.880442           1.68982   1.74883            0.883929         0.914798                    0.0171044       0.0352453                  68.9819   74.8832\n",
       "    3        0.0300045                   0.864254           1.6311    1.71018            0.853211         0.894578                    0.0160677       0.0513131                  63.1095   71.0178\n",
       "    4        0.0400362                   0.852258           1.72226   1.71321            0.900901         0.896163                    0.0172771       0.0685902                  72.2265   71.3206\n",
       "    5        0.0500678                   0.84136            1.73949   1.71847            0.90991          0.898917                    0.0174499       0.0860401                  73.9487   71.8472\n",
       "    6        0.100136                    0.792052           1.60115   1.65981            0.837545         0.868231                    0.0801659       0.166206                   60.1147   65.9809\n",
       "    7        0.150203                    0.748608           1.47692   1.59885            0.772563         0.836342                    0.0739461       0.240152                   47.692    59.8846\n",
       "    8        0.200271                    0.710225           1.4079    1.55111            0.736462         0.811372                    0.0704907       0.310643                   40.7905   55.1111\n",
       "    9        0.300045                    0.653055           1.35586   1.48618            0.709239         0.77741                     0.13528         0.445923                   35.5862   48.6185\n",
       "    10       0.400904                    0.579179           1.17683   1.40836            0.615591         0.7367                      0.118694        0.564616                   17.6835   40.8359\n",
       "    11       0.500045                    0.507929           1.09266   1.34577            0.571559         0.703958                    0.108328        0.672944                   9.26569   34.5766\n",
       "    12       0.600271                    0.44367            0.884319  1.26872            0.462579         0.663656                    0.0886317       0.761576                   -11.5681  26.8719\n",
       "    13       0.700045                    0.386446           0.749794  1.19476            0.39221          0.624968                    0.07481         0.836386                   -25.0206  19.4759\n",
       "    14       0.80009                     0.322995           0.6476    1.12634            0.338753         0.589179                    0.0647892       0.901175                   -35.24    12.6341\n",
       "    15       0.899955                    0.256694           0.59341   1.0672             0.310407         0.558245                    0.0592605       0.960435                   -40.659   6.7204\n",
       "    16       1                           0.0676948          0.395467  1                  0.206865         0.523091                    0.0395646       1                          -60.4533  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# GBM Performance\n",
    "data_gbm2.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsBinomial: deeplearning\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.2170978543303148\n",
      "R^2: 0.12975256668292567\n",
      "LogLoss: 0.6237081078318155\n",
      "AUC: 0.7209628233723193\n",
      "Gini: 0.44192564674463863\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.29878787453148525: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>2031.0</td>\n",
       "<td>3246.0</td>\n",
       "<td>0.6151</td>\n",
       "<td> (3246.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>789.0</td>\n",
       "<td>4999.0</td>\n",
       "<td>0.1363</td>\n",
       "<td> (789.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2820.0</td>\n",
       "<td>8245.0</td>\n",
       "<td>0.3647</td>\n",
       "<td> (4035.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     2031  3246   0.6151   (3246.0/5277.0)\n",
       "YES    789   4999   0.1363   (789.0/5788.0)\n",
       "Total  2820  8245   0.3647   (4035.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.2987879</td>\n",
       "<td>0.7124635</td>\n",
       "<td>310.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1231821</td>\n",
       "<td>0.8465415</td>\n",
       "<td>392.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.4613361</td>\n",
       "<td>0.6840484</td>\n",
       "<td>220.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.4008738</td>\n",
       "<td>0.6666968</td>\n",
       "<td>254.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9920392</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0999376</td>\n",
       "<td>1.0</td>\n",
       "<td>397.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9920392</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.4613361</td>\n",
       "<td>0.3322544</td>\n",
       "<td>220.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.4315932</td>\n",
       "<td>0.6611711</td>\n",
       "<td>236.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.298788     0.712463  310\n",
       "max f2                      0.123182     0.846541  392\n",
       "max f0point5                0.461336     0.684048  220\n",
       "max accuracy                0.400874     0.666697  254\n",
       "max precision               0.992039     1         0\n",
       "max recall                  0.0999376    1         397\n",
       "max specificity             0.992039     1         0\n",
       "max absolute_MCC            0.461336     0.332254  220\n",
       "max min_per_class_accuracy  0.431593     0.661171  236"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0101220</td>\n",
       "<td>0.9283660</td>\n",
       "<td>1.8093006</td>\n",
       "<td>1.8093006</td>\n",
       "<td>0.9464286</td>\n",
       "<td>0.9464286</td>\n",
       "<td>0.0183138</td>\n",
       "<td>0.0183138</td>\n",
       "<td>80.9300647</td>\n",
       "<td>80.9300647</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0200633</td>\n",
       "<td>0.8904271</td>\n",
       "<td>1.7205425</td>\n",
       "<td>1.7653214</td>\n",
       "<td>0.9</td>\n",
       "<td>0.9234234</td>\n",
       "<td>0.0171044</td>\n",
       "<td>0.0354181</td>\n",
       "<td>72.0542502</td>\n",
       "<td>76.5321386</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.8634937</td>\n",
       "<td>1.6162672</td>\n",
       "<td>1.7159360</td>\n",
       "<td>0.8454545</td>\n",
       "<td>0.8975904</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0514858</td>\n",
       "<td>61.6267199</td>\n",
       "<td>71.5935962</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.8469819</td>\n",
       "<td>1.6705968</td>\n",
       "<td>1.7045756</td>\n",
       "<td>0.8738739</td>\n",
       "<td>0.8916479</td>\n",
       "<td>0.0167588</td>\n",
       "<td>0.0682446</td>\n",
       "<td>67.0596823</td>\n",
       "<td>70.4575591</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.8255683</td>\n",
       "<td>1.6017062</td>\n",
       "<td>1.6839646</td>\n",
       "<td>0.8378378</td>\n",
       "<td>0.8808664</td>\n",
       "<td>0.0160677</td>\n",
       "<td>0.0843124</td>\n",
       "<td>60.1706233</td>\n",
       "<td>68.3964583</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1000452</td>\n",
       "<td>0.7519715</td>\n",
       "<td>1.5694722</td>\n",
       "<td>1.6267701</td>\n",
       "<td>0.8209765</td>\n",
       "<td>0.8509485</td>\n",
       "<td>0.0784381</td>\n",
       "<td>0.1627505</td>\n",
       "<td>56.9472164</td>\n",
       "<td>62.6770086</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.6936521</td>\n",
       "<td>1.4346497</td>\n",
       "<td>1.5627685</td>\n",
       "<td>0.7504521</td>\n",
       "<td>0.8174699</td>\n",
       "<td>0.0717001</td>\n",
       "<td>0.2344506</td>\n",
       "<td>43.4649665</td>\n",
       "<td>56.2768524</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2</td>\n",
       "<td>0.6494949</td>\n",
       "<td>1.3413110</td>\n",
       "<td>1.5074292</td>\n",
       "<td>0.7016275</td>\n",
       "<td>0.7885224</td>\n",
       "<td>0.0670352</td>\n",
       "<td>0.3014858</td>\n",
       "<td>34.1311012</td>\n",
       "<td>50.7429164</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.5722457</td>\n",
       "<td>1.3245570</td>\n",
       "<td>1.4464534</td>\n",
       "<td>0.6928636</td>\n",
       "<td>0.7566265</td>\n",
       "<td>0.1325155</td>\n",
       "<td>0.4340014</td>\n",
       "<td>32.4556960</td>\n",
       "<td>44.6453402</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.5049965</td>\n",
       "<td>1.1719186</td>\n",
       "<td>1.3778507</td>\n",
       "<td>0.6130199</td>\n",
       "<td>0.7207411</td>\n",
       "<td>0.1171389</td>\n",
       "<td>0.5511403</td>\n",
       "<td>17.1918642</td>\n",
       "<td>37.7850726</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5000452</td>\n",
       "<td>0.4363803</td>\n",
       "<td>1.0447940</td>\n",
       "<td>1.3112153</td>\n",
       "<td>0.5465221</td>\n",
       "<td>0.6858847</td>\n",
       "<td>0.1045266</td>\n",
       "<td>0.6556669</td>\n",
       "<td>4.4793951</td>\n",
       "<td>31.1215293</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6</td>\n",
       "<td>0.3773007</td>\n",
       "<td>0.9558569</td>\n",
       "<td>1.2520157</td>\n",
       "<td>0.5</td>\n",
       "<td>0.6549179</td>\n",
       "<td>0.0955425</td>\n",
       "<td>0.7512094</td>\n",
       "<td>-4.4143055</td>\n",
       "<td>25.2015665</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.6999548</td>\n",
       "<td>0.3224394</td>\n",
       "<td>0.7760936</td>\n",
       "<td>1.1840531</td>\n",
       "<td>0.4059675</td>\n",
       "<td>0.6193673</td>\n",
       "<td>0.0775743</td>\n",
       "<td>0.8287837</td>\n",
       "<td>-22.3906386</td>\n",
       "<td>18.4053135</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8</td>\n",
       "<td>0.2690563</td>\n",
       "<td>0.6907729</td>\n",
       "<td>1.1223652</td>\n",
       "<td>0.3613369</td>\n",
       "<td>0.5870990</td>\n",
       "<td>0.0691085</td>\n",
       "<td>0.8978922</td>\n",
       "<td>-30.9227140</td>\n",
       "<td>12.2365238</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.9000452</td>\n",
       "<td>0.2048145</td>\n",
       "<td>0.5871569</td>\n",
       "<td>1.0628738</td>\n",
       "<td>0.3071364</td>\n",
       "<td>0.5559795</td>\n",
       "<td>0.0587422</td>\n",
       "<td>0.9566344</td>\n",
       "<td>-41.2843069</td>\n",
       "<td>6.2873764</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0899145</td>\n",
       "<td>0.4338519</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2269439</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0433656</td>\n",
       "<td>1.0</td>\n",
       "<td>-56.6148113</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.010122                    0.928366           1.8093    1.8093             0.946429         0.946429                    0.0183138       0.0183138                  80.9301   80.9301\n",
       "    2        0.0200633                   0.890427           1.72054   1.76532            0.9              0.923423                    0.0171044       0.0354181                  72.0543   76.5321\n",
       "    3        0.0300045                   0.863494           1.61627   1.71594            0.845455         0.89759                     0.0160677       0.0514858                  61.6267   71.5936\n",
       "    4        0.0400362                   0.846982           1.6706    1.70458            0.873874         0.891648                    0.0167588       0.0682446                  67.0597   70.4576\n",
       "    5        0.0500678                   0.825568           1.60171   1.68396            0.837838         0.880866                    0.0160677       0.0843124                  60.1706   68.3965\n",
       "    6        0.100045                    0.751971           1.56947   1.62677            0.820976         0.850949                    0.0784381       0.162751                   56.9472   62.677\n",
       "    7        0.150023                    0.693652           1.43465   1.56277            0.750452         0.81747                     0.0717001       0.234451                   43.465    56.2769\n",
       "    8        0.2                         0.649495           1.34131   1.50743            0.701627         0.788522                    0.0670352       0.301486                   34.1311   50.7429\n",
       "    9        0.300045                    0.572246           1.32456   1.44645            0.692864         0.756627                    0.132516        0.434001                   32.4557   44.6453\n",
       "    10       0.4                         0.504997           1.17192   1.37785            0.61302          0.720741                    0.117139        0.55114                    17.1919   37.7851\n",
       "    11       0.500045                    0.43638            1.04479   1.31122            0.546522         0.685885                    0.104527        0.655667                   4.4794    31.1215\n",
       "    12       0.6                         0.377301           0.955857  1.25202            0.5              0.654918                    0.0955425       0.751209                   -4.41431  25.2016\n",
       "    13       0.699955                    0.322439           0.776094  1.18405            0.405967         0.619367                    0.0775743       0.828784                   -22.3906  18.4053\n",
       "    14       0.8                         0.269056           0.690773  1.12237            0.361337         0.587099                    0.0691085       0.897892                   -30.9227  12.2365\n",
       "    15       0.900045                    0.204814           0.587157  1.06287            0.307136         0.55598                     0.0587422       0.956634                   -41.2843  6.28738\n",
       "    16       1                           0.0899145          0.433852  1                  0.226944         0.523091                    0.0433656       1                          -56.6148  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Deep Learning Performance\n",
    "data_dl.model_performance(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsBinomial: naivebayes\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 0.22952227413516735\n",
      "R^2: 0.07994866844091097\n",
      "LogLoss: 0.6604736519444419\n",
      "AUC: 0.6926305809501246\n",
      "Gini: 0.3852611619002493\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.2227661773519952: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>NO</b></td>\n",
       "<td><b>YES</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>NO</td>\n",
       "<td>1169.0</td>\n",
       "<td>4108.0</td>\n",
       "<td>0.7785</td>\n",
       "<td> (4108.0/5277.0)</td></tr>\n",
       "<tr><td>YES</td>\n",
       "<td>430.0</td>\n",
       "<td>5358.0</td>\n",
       "<td>0.0743</td>\n",
       "<td> (430.0/5788.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>1599.0</td>\n",
       "<td>9466.0</td>\n",
       "<td>0.4101</td>\n",
       "<td> (4538.0/11065.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       NO    YES    Error    Rate\n",
       "-----  ----  -----  -------  ----------------\n",
       "NO     1169  4108   0.7785   (4108.0/5277.0)\n",
       "YES    430   5358   0.0743   (430.0/5788.0)\n",
       "Total  1599  9466   0.4101   (4538.0/11065.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.2227662</td>\n",
       "<td>0.7025043</td>\n",
       "<td>334.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.0499286</td>\n",
       "<td>0.8468321</td>\n",
       "<td>390.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5720444</td>\n",
       "<td>0.6629061</td>\n",
       "<td>187.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5720444</td>\n",
       "<td>0.6419340</td>\n",
       "<td>187.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9972780</td>\n",
       "<td>0.9807692</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0129773</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9972780</td>\n",
       "<td>0.9998105</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5720444</td>\n",
       "<td>0.2928394</td>\n",
       "<td>187.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.5013849</td>\n",
       "<td>0.6371044</td>\n",
       "<td>218.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.222766     0.702504  334\n",
       "max f2                      0.0499286    0.846832  390\n",
       "max f0point5                0.572044     0.662906  187\n",
       "max accuracy                0.572044     0.641934  187\n",
       "max precision               0.997278     0.980769  0\n",
       "max recall                  0.0129773    1         399\n",
       "max specificity             0.997278     0.99981   0\n",
       "max absolute_MCC            0.572044     0.292839  187\n",
       "max min_per_class_accuracy  0.501385     0.637104  218"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 52.31 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.0100316</td>\n",
       "<td>0.9867983</td>\n",
       "<td>1.8600459</td>\n",
       "<td>1.8600459</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.9729730</td>\n",
       "<td>0.0186593</td>\n",
       "<td>0.0186593</td>\n",
       "<td>86.0045948</td>\n",
       "<td>86.0045948</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.0200633</td>\n",
       "<td>0.9728750</td>\n",
       "<td>1.7222648</td>\n",
       "<td>1.7911554</td>\n",
       "<td>0.9009009</td>\n",
       "<td>0.9369369</td>\n",
       "<td>0.0172771</td>\n",
       "<td>0.0359364</td>\n",
       "<td>72.2264766</td>\n",
       "<td>79.1155357</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.0300045</td>\n",
       "<td>0.9608781</td>\n",
       "<td>1.5815088</td>\n",
       "<td>1.7216941</td>\n",
       "<td>0.8272727</td>\n",
       "<td>0.9006024</td>\n",
       "<td>0.0157222</td>\n",
       "<td>0.0516586</td>\n",
       "<td>58.1508764</td>\n",
       "<td>72.1694137</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.0400362</td>\n",
       "<td>0.9452227</td>\n",
       "<td>1.4122571</td>\n",
       "<td>1.6441603</td>\n",
       "<td>0.7387387</td>\n",
       "<td>0.8600451</td>\n",
       "<td>0.0141672</td>\n",
       "<td>0.0658258</td>\n",
       "<td>41.2257109</td>\n",
       "<td>64.4160254</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.0500678</td>\n",
       "<td>0.9312264</td>\n",
       "<td>1.5500383</td>\n",
       "<td>1.6253019</td>\n",
       "<td>0.8108108</td>\n",
       "<td>0.8501805</td>\n",
       "<td>0.0155494</td>\n",
       "<td>0.0813753</td>\n",
       "<td>55.0038290</td>\n",
       "<td>62.5301882</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.1000452</td>\n",
       "<td>0.8739324</td>\n",
       "<td>1.5141604</td>\n",
       "<td>1.5697813</td>\n",
       "<td>0.7920434</td>\n",
       "<td>0.8211382</td>\n",
       "<td>0.0756738</td>\n",
       "<td>0.1570491</td>\n",
       "<td>51.4160369</td>\n",
       "<td>56.9781325</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.1500226</td>\n",
       "<td>0.8246779</td>\n",
       "<td>1.4173647</td>\n",
       "<td>1.5190064</td>\n",
       "<td>0.7414105</td>\n",
       "<td>0.7945783</td>\n",
       "<td>0.0708362</td>\n",
       "<td>0.2278853</td>\n",
       "<td>41.7364729</td>\n",
       "<td>51.9006399</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.2</td>\n",
       "<td>0.7696958</td>\n",
       "<td>1.2894562</td>\n",
       "<td>1.4616448</td>\n",
       "<td>0.6745027</td>\n",
       "<td>0.7645730</td>\n",
       "<td>0.0644437</td>\n",
       "<td>0.2923290</td>\n",
       "<td>28.9456205</td>\n",
       "<td>46.1644782</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.3000452</td>\n",
       "<td>0.6837439</td>\n",
       "<td>1.2882914</td>\n",
       "<td>1.4038429</td>\n",
       "<td>0.6738934</td>\n",
       "<td>0.7343373</td>\n",
       "<td>0.1288874</td>\n",
       "<td>0.4212163</td>\n",
       "<td>28.8291384</td>\n",
       "<td>40.3842911</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4</td>\n",
       "<td>0.6035968</td>\n",
       "<td>1.1338918</td>\n",
       "<td>1.3363856</td>\n",
       "<td>0.5931284</td>\n",
       "<td>0.6990511</td>\n",
       "<td>0.1133379</td>\n",
       "<td>0.5345543</td>\n",
       "<td>13.3891783</td>\n",
       "<td>33.6385625</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.5002259</td>\n",
       "<td>0.5062851</td>\n",
       "<td>0.9705094</td>\n",
       "<td>1.2630782</td>\n",
       "<td>0.5076646</td>\n",
       "<td>0.6607046</td>\n",
       "<td>0.0972702</td>\n",
       "<td>0.6318245</td>\n",
       "<td>-2.9490604</td>\n",
       "<td>26.3078175</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.6</td>\n",
       "<td>0.4147244</td>\n",
       "<td>0.8848603</td>\n",
       "<td>1.2001843</td>\n",
       "<td>0.4628623</td>\n",
       "<td>0.6278054</td>\n",
       "<td>0.0882861</td>\n",
       "<td>0.7201106</td>\n",
       "<td>-11.5139676</td>\n",
       "<td>20.0184289</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.6999548</td>\n",
       "<td>0.3256296</td>\n",
       "<td>0.8711608</td>\n",
       "<td>1.1531991</td>\n",
       "<td>0.4556962</td>\n",
       "<td>0.6032279</td>\n",
       "<td>0.0870767</td>\n",
       "<td>0.8071873</td>\n",
       "<td>-12.8839240</td>\n",
       "<td>15.3199135</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.8</td>\n",
       "<td>0.2599485</td>\n",
       "<td>0.7650309</td>\n",
       "<td>1.1046562</td>\n",
       "<td>0.4001807</td>\n",
       "<td>0.5778355</td>\n",
       "<td>0.0765377</td>\n",
       "<td>0.8837249</td>\n",
       "<td>-23.4969057</td>\n",
       "<td>10.4656185</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.9000452</td>\n",
       "<td>0.1793016</td>\n",
       "<td>0.6786843</td>\n",
       "<td>1.0573070</td>\n",
       "<td>0.3550136</td>\n",
       "<td>0.5530676</td>\n",
       "<td>0.0678991</td>\n",
       "<td>0.9516240</td>\n",
       "<td>-32.1315665</td>\n",
       "<td>5.7306970</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0098785</td>\n",
       "<td>0.4839782</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2531646</td>\n",
       "<td>0.5230908</td>\n",
       "<td>0.0483760</td>\n",
       "<td>1.0</td>\n",
       "<td>-51.6021800</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    gain      cumulative_gain\n",
       "--  -------  --------------------------  -----------------  --------  -----------------  ---------------  --------------------------  --------------  -------------------------  --------  -----------------\n",
       "    1        0.0100316                   0.986798           1.86005   1.86005            0.972973         0.972973                    0.0186593       0.0186593                  86.0046   86.0046\n",
       "    2        0.0200633                   0.972875           1.72226   1.79116            0.900901         0.936937                    0.0172771       0.0359364                  72.2265   79.1155\n",
       "    3        0.0300045                   0.960878           1.58151   1.72169            0.827273         0.900602                    0.0157222       0.0516586                  58.1509   72.1694\n",
       "    4        0.0400362                   0.945223           1.41226   1.64416            0.738739         0.860045                    0.0141672       0.0658258                  41.2257   64.416\n",
       "    5        0.0500678                   0.931226           1.55004   1.6253             0.810811         0.850181                    0.0155494       0.0813753                  55.0038   62.5302\n",
       "    6        0.100045                    0.873932           1.51416   1.56978            0.792043         0.821138                    0.0756738       0.157049                   51.416    56.9781\n",
       "    7        0.150023                    0.824678           1.41736   1.51901            0.74141          0.794578                    0.0708362       0.227885                   41.7365   51.9006\n",
       "    8        0.2                         0.769696           1.28946   1.46164            0.674503         0.764573                    0.0644437       0.292329                   28.9456   46.1645\n",
       "    9        0.300045                    0.683744           1.28829   1.40384            0.673893         0.734337                    0.128887        0.421216                   28.8291   40.3843\n",
       "    10       0.4                         0.603597           1.13389   1.33639            0.593128         0.699051                    0.113338        0.534554                   13.3892   33.6386\n",
       "    11       0.500226                    0.506285           0.970509  1.26308            0.507665         0.660705                    0.0972702       0.631824                   -2.94906  26.3078\n",
       "    12       0.6                         0.414724           0.88486   1.20018            0.462862         0.627805                    0.0882861       0.720111                   -11.514   20.0184\n",
       "    13       0.699955                    0.32563            0.871161  1.1532             0.455696         0.603228                    0.0870767       0.807187                   -12.8839  15.3199\n",
       "    14       0.8                         0.259948           0.765031  1.10466            0.400181         0.577836                    0.0765377       0.883725                   -23.4969  10.4656\n",
       "    15       0.900045                    0.179302           0.678684  1.05731            0.355014         0.553068                    0.0678991       0.951624                   -32.1316  5.7307\n",
       "    16       1                           0.00987846         0.483978  1                  0.253165         0.523091                    0.048376        1                          -51.6022  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Naive Bayes Performance\n",
    "data_nb.model_performance(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GLM Coefficient Magnitudes / DRF, GBM & Deep Learning Variable Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficient Magnitudes:\n",
      "\n",
      "| Predictor                 |   Normalized Coefficient |\n",
      "|---------------------------+--------------------------|\n",
      "| Year.2008                 |               2.19858    |\n",
      "| Origin.MDW                |               1.67302    |\n",
      "| Origin.HPN                |               1.56693    |\n",
      "| Year.2003                 |               1.55654    |\n",
      "| Origin.LIH                |               1.49892    |\n",
      "| Year.2007                 |               1.46806    |\n",
      "| Dest.LYH                  |               1.25683    |\n",
      "| UniqueCarrier.HP          |               1.19937    |\n",
      "| Dest.HTS                  |               1.15907    |\n",
      "| Origin.TLH                |               1.13519    |\n",
      "| Origin.LEX                |               1.08467    |\n",
      "| Origin.HNL                |               1.04093    |\n",
      "| Origin.CHO                |               1.02275    |\n",
      "| Origin.ATL                |               0.999746   |\n",
      "| Year.2001                 |               0.982825   |\n",
      "| Origin.PSP                |               0.963531   |\n",
      "| Year.2002                 |               0.963237   |\n",
      "| UniqueCarrier.TW          |               0.960336   |\n",
      "| Origin.ERI                |               0.955332   |\n",
      "| Origin.STX                |               0.947432   |\n",
      "| Origin.CAE                |               0.869595   |\n",
      "| Year.2004                 |               0.860763   |\n",
      "| Origin.ORD                |               0.839797   |\n",
      "| Origin.PBI                |               0.800896   |\n",
      "| Year.2006                 |               0.772002   |\n",
      "| Dest.CHO                  |               0.76052    |\n",
      "| Origin.LYH                |               0.76052    |\n",
      "| Origin.MYR                |               0.757717   |\n",
      "| Origin.IAH                |               0.747385   |\n",
      "| Dest.FLL                  |               0.741809   |\n",
      "| Dest.AVL                  |               0.734992   |\n",
      "| Year.1994                 |               0.725507   |\n",
      "| Dest.DAY                  |               0.718767   |\n",
      "| Origin.OGG                |               0.705689   |\n",
      "| Dest.PSP                  |               0.692691   |\n",
      "| Origin.CMH                |               0.687228   |\n",
      "| Origin.STL                |               0.682812   |\n",
      "| Dest.ICT                  |               0.672662   |\n",
      "| Origin.ALB                |               0.640848   |\n",
      "| Origin.ROA                |               0.633851   |\n",
      "| Year.1996                 |               0.631063   |\n",
      "| Origin.TRI                |               0.601505   |\n",
      "| Origin.SYR                |               0.596296   |\n",
      "| Dest.PNS                  |               0.588903   |\n",
      "| Origin.AUS                |               0.582793   |\n",
      "| Dest.GSO                  |               0.579513   |\n",
      "| Dest.GEG                  |               0.571285   |\n",
      "| Origin.BTV                |               0.568473   |\n",
      "| Origin.LAX                |               0.566536   |\n",
      "| Origin.MIA                |               0.566439   |\n",
      "| Origin.OMA                |               0.558284   |\n",
      "| Origin.ACY                |               0.557115   |\n",
      "| Year.1997                 |               0.552412   |\n",
      "| Origin.PIT                |               0.54741    |\n",
      "| Year.1990                 |               0.539981   |\n",
      "| Dest.FAY                  |               0.53628    |\n",
      "| Origin.CRW                |               0.527206   |\n",
      "| Dest.OGG                  |               0.524551   |\n",
      "| Dest.SFO                  |               0.522489   |\n",
      "| Origin.CRP                |               0.517442   |\n",
      "| Origin.EYW                |               0.512461   |\n",
      "| Dest.IAH                  |               0.511763   |\n",
      "| UniqueCarrier.PI          |               0.50913    |\n",
      "| Dest.SLC                  |               0.509118   |\n",
      "| Origin.FLL                |               0.503688   |\n",
      "| UniqueCarrier.WN          |               0.498006   |\n",
      "| Origin.OKC                |               0.495998   |\n",
      "| Dest.ISP                  |               0.495776   |\n",
      "| Origin.MSY                |               0.489918   |\n",
      "| Year.2005                 |               0.48681    |\n",
      "| Origin.MCO                |               0.477058   |\n",
      "| Origin.IND                |               0.473817   |\n",
      "| Origin.RNO                |               0.469941   |\n",
      "| Dest.BGM                  |               0.466168   |\n",
      "| Dest.IND                  |               0.457781   |\n",
      "| Origin.PHL                |               0.456633   |\n",
      "| Dest.UCA                  |               0.452046   |\n",
      "| Origin.GSO                |               0.45192    |\n",
      "| Origin.PWM                |               0.437979   |\n",
      "| Dest.TPA                  |               0.431338   |\n",
      "| Origin.SRQ                |               0.422504   |\n",
      "| Origin.DAY                |               0.413958   |\n",
      "| Dest.CAK                  |               0.413293   |\n",
      "| Origin.DFW                |               0.412907   |\n",
      "| Origin.SAV                |               0.410809   |\n",
      "| UniqueCarrier.CO          |               0.40659    |\n",
      "| Dest.BDL                  |               0.406095   |\n",
      "| Dest.CAE                  |               0.401772   |\n",
      "| Dest.COS                  |               0.397441   |\n",
      "| Origin.OAK                |               0.381577   |\n",
      "| Origin.BOI                |               0.380043   |\n",
      "| Origin.JAX                |               0.376318   |\n",
      "| Dest.PBI                  |               0.373462   |\n",
      "| Dest.SDF                  |               0.372169   |\n",
      "| Year.1995                 |               0.367096   |\n",
      "| Dest.BUF                  |               0.36642    |\n",
      "| Dest.SWF                  |               0.36335    |\n",
      "| Origin.ELP                |               0.356208   |\n",
      "| Dest.CLE                  |               0.355224   |\n",
      "| Origin.LGA                |               0.353049   |\n",
      "| Dest.JAX                  |               0.352995   |\n",
      "| Dest.ALB                  |               0.348283   |\n",
      "| Origin.BUF                |               0.346038   |\n",
      "| Origin.MCI                |               0.345448   |\n",
      "| Origin.BOS                |               0.343105   |\n",
      "| Origin.BDL                |               0.340712   |\n",
      "| Dest.BTV                  |               0.332514   |\n",
      "| Origin.ROC                |               0.330604   |\n",
      "| Origin.MSP                |               0.327787   |\n",
      "| Origin.JFK                |               0.321378   |\n",
      "| Dest.MCO                  |               0.319434   |\n",
      "| Dest.ABQ                  |               0.31625    |\n",
      "| Dest.CMH                  |               0.31476    |\n",
      "| Dest.LIH                  |               0.314168   |\n",
      "| Dest.LBB                  |               0.308055   |\n",
      "| UniqueCarrier.US          |               0.306931   |\n",
      "| Origin.BWI                |               0.302733   |\n",
      "| Dest.STL                  |               0.299111   |\n",
      "| Origin.SLC                |               0.298382   |\n",
      "| Origin.SAN                |               0.296723   |\n",
      "| Origin.CLT                |               0.295934   |\n",
      "| Origin.TUS                |               0.295071   |\n",
      "| Origin.BHM                |               0.293635   |\n",
      "| Dest.KOA                  |               0.289065   |\n",
      "| Year.1992                 |               0.2847     |\n",
      "| Dest.TUL                  |               0.283613   |\n",
      "| Origin.UCA                |               0.268208   |\n",
      "| DayOfWeek.5               |               0.263935   |\n",
      "| Origin.CLE                |               0.261628   |\n",
      "| Year.1991                 |               0.252562   |\n",
      "| Origin.COS                |               0.249223   |\n",
      "| Dest.CLT                  |               0.249012   |\n",
      "| Origin.BUR                |               0.245153   |\n",
      "| Dest.FAT                  |               0.242143   |\n",
      "| Month.10                  |               0.241364   |\n",
      "| Year.1987                 |               0.241364   |\n",
      "| Dest.ROA                  |               0.240329   |\n",
      "| Origin.PHF                |               0.239999   |\n",
      "| Dest.OMA                  |               0.239768   |\n",
      "| Origin.SMF                |               0.234955   |\n",
      "| Dest.SEA                  |               0.229919   |\n",
      "| Dest.HRL                  |               0.218484   |\n",
      "| Origin.DEN                |               0.216983   |\n",
      "| Year.1999                 |               0.211729   |\n",
      "| Dest.PDX                  |               0.210602   |\n",
      "| Dest.LAS                  |               0.206601   |\n",
      "| Dest.MSP                  |               0.204311   |\n",
      "| Origin.IAD                |               0.197859   |\n",
      "| Dest.BUR                  |               0.197227   |\n",
      "| Year.2000                 |               0.193379   |\n",
      "| Origin.SFO                |               0.190741   |\n",
      "| Origin.BGM                |               0.189911   |\n",
      "| Origin.SWF                |               0.189525   |\n",
      "| Dest.IAD                  |               0.18846    |\n",
      "| Dest.SAT                  |               0.185439   |\n",
      "| Dest.ROC                  |               0.183779   |\n",
      "| Dest.LAX                  |               0.180429   |\n",
      "| Origin.TPA                |               0.179862   |\n",
      "| Dest.PHL                  |               0.170191   |\n",
      "| Distance                  |               0.167585   |\n",
      "| Origin.DTW                |               0.159638   |\n",
      "| Dest.BNA                  |               0.159266   |\n",
      "| DayOfWeek.4               |               0.157277   |\n",
      "| Dest.OAK                  |               0.15125    |\n",
      "| Dest.FNT                  |               0.150786   |\n",
      "| Origin.PDX                |               0.15073    |\n",
      "| Origin.RDU                |               0.144973   |\n",
      "| Origin.ORF                |               0.144369   |\n",
      "| Year.1989                 |               0.144251   |\n",
      "| Dest.BWI                  |               0.143257   |\n",
      "| Dest.ELP                  |               0.142565   |\n",
      "| Dest.MDW                  |               0.134653   |\n",
      "| UniqueCarrier.AA          |               0.133115   |\n",
      "| Year.1993                 |               0.130076   |\n",
      "| Dest.PHX                  |               0.129837   |\n",
      "| Dest.SAN                  |               0.129307   |\n",
      "| DayOfWeek.2               |               0.127522   |\n",
      "| Origin.JAN                |               0.125264   |\n",
      "| Origin.MKE                |               0.124907   |\n",
      "| Origin.DAL                |               0.121417   |\n",
      "| Dest.EWR                  |               0.121082   |\n",
      "| Month.1                   |               0.117726   |\n",
      "| Dest.RNO                  |               0.113732   |\n",
      "| Origin.SJC                |               0.113654   |\n",
      "| Dest.MAF                  |               0.112554   |\n",
      "| Dest.ORD                  |               0.111985   |\n",
      "| DayOfWeek.6               |               0.109718   |\n",
      "| Dest.SBN                  |               0.109414   |\n",
      "| Origin.ISP                |               0.105969   |\n",
      "| UniqueCarrier.UA          |               0.0936254  |\n",
      "| Origin.RIC                |               0.0921766  |\n",
      "| Origin.PVD                |               0.0818253  |\n",
      "| Dest.ORF                  |               0.07756    |\n",
      "| Dest.MIA                  |               0.0775558  |\n",
      "| Origin.CHS                |               0.072934   |\n",
      "| Origin.LAS                |               0.0687986  |\n",
      "| Dest.ABE                  |               0.0664163  |\n",
      "| Origin.EWR                |               0.0642849  |\n",
      "| Dest.DTW                  |               0.0633446  |\n",
      "| Dest.HNL                  |               0.0610877  |\n",
      "| Dest.ACY                  |               0.0584843  |\n",
      "| Intercept                 |               0.0583759  |\n",
      "| Dest.RSW                  |               0.054807   |\n",
      "| Dest.DEN                  |               0.0531848  |\n",
      "| Dest.TUS                  |               0.0503133  |\n",
      "| Dest.SMF                  |               0.0495967  |\n",
      "| Dest.BOS                  |               0.0492589  |\n",
      "| Dest.MKE                  |               0.0488848  |\n",
      "| Dest.LGA                  |               0.045868   |\n",
      "| Origin.MEM                |               0.0415562  |\n",
      "| Origin.DCA                |               0.0409492  |\n",
      "| Origin.SNA                |               0.0380913  |\n",
      "| Dest.MDT                  |               0.0359403  |\n",
      "| DayOfWeek.7               |               0.0330296  |\n",
      "| Dest.DFW                  |               0.032886   |\n",
      "| Dest.MCI                  |               0.0325893  |\n",
      "| Dest.SNA                  |               0.0310822  |\n",
      "| Year.1998                 |               0.0305543  |\n",
      "| DayOfWeek.3               |               0.0289725  |\n",
      "| Year.1988                 |               0.0255815  |\n",
      "| FlightNum                 |               0.0238619  |\n",
      "| Dest.PIT                  |               0.0205454  |\n",
      "| Origin.ONT                |               0.0203642  |\n",
      "| Origin.SJU                |               0.0172619  |\n",
      "| Origin.MDT                |               0.0171563  |\n",
      "| Dest.BOI                  |               0.0171016  |\n",
      "| Dest.MSY                  |               0.010624   |\n",
      "| Dest.DAL                  |               0.00981252 |\n",
      "| Dest.ATL                  |               0.00782332 |\n",
      "| UniqueCarrier.DL          |               0.00670187 |\n",
      "| Dest.PVD                  |               0.00252622 |\n",
      "| Dest.JAN                  |               0.00252588 |\n",
      "| Dest.SJC                  |               0.00177487 |\n",
      "| Dest.SYR                  |               0.00165583 |\n",
      "| Dest.AUS                  |               0.00048016 |\n",
      "| Origin.KOA                |               0          |\n",
      "| Dest.ORH                  |               0          |\n",
      "| Month.missing(NA)         |               0          |\n",
      "| Dest.MYR                  |               0          |\n",
      "| Origin.LBB                |               0          |\n",
      "| Dest.BHM                  |               0          |\n",
      "| Origin.ABE                |               0          |\n",
      "| DayOfWeek.1               |               0          |\n",
      "| Origin.LIT                |               0          |\n",
      "| Dest.AVP                  |               0          |\n",
      "| Dest.SRQ                  |               0          |\n",
      "| Dest.ILM                  |               0          |\n",
      "| Origin.GRR                |               0          |\n",
      "| Dest.PHF                  |               0          |\n",
      "| Origin.MLB                |               0          |\n",
      "| Origin.DSM                |               0          |\n",
      "| Origin.SAT                |               0          |\n",
      "| Dest.RDU                  |               0          |\n",
      "| Dest.HPN                  |               0          |\n",
      "| Dest.JFK                  |               0          |\n",
      "| Dest.CRP                  |               0          |\n",
      "| Dest.EUG                  |               0          |\n",
      "| Dest.OKC                  |               0          |\n",
      "| Origin.MHT                |               0          |\n",
      "| Dest.STT                  |               0          |\n",
      "| Origin.AMA                |               0          |\n",
      "| Origin.TYS                |               0          |\n",
      "| Dest.DSM                  |               0          |\n",
      "| Origin.EGE                |               0          |\n",
      "| Dest.GRR                  |               0          |\n",
      "| Dest.LIT                  |               0          |\n",
      "| Origin.LAN                |               0          |\n",
      "| Dest.LEX                  |               0          |\n",
      "| Dest.ANC                  |               0          |\n",
      "| Dest.MRY                  |               0          |\n",
      "| Dest.CHS                  |               0          |\n",
      "| UniqueCarrier.missing(NA) |               0          |\n",
      "| Dest.AMA                  |               0          |\n",
      "| Origin.MFR                |               0          |\n",
      "| Origin.SEA                |               0          |\n",
      "| Dest.OAJ                  |               0          |\n",
      "| Origin.GNV                |               0          |\n",
      "| Origin.SCK                |               0          |\n",
      "| Origin.HOU                |               0          |\n",
      "| Dest.ERI                  |               0          |\n",
      "| Dest.GSP                  |               0          |\n",
      "| Dest.DCA                  |               0          |\n",
      "| Origin.CVG                |               0          |\n",
      "| Dest.ELM                  |               0          |\n",
      "| Dest.MHT                  |               0          |\n",
      "| Origin.TUL                |               0          |\n",
      "| Origin.ICT                |               0          |\n",
      "| Origin.MRY                |               0          |\n",
      "| Year.missing(NA)          |               0          |\n",
      "| Origin.GEG                |               0          |\n",
      "| Dest.RIC                  |               0          |\n",
      "| Dest.HOU                  |               0          |\n",
      "| Dest.ONT                  |               0          |\n",
      "| UniqueCarrier.PS          |               0          |\n",
      "| Dest.CVG                  |               0          |\n",
      "| Origin.missing(NA)        |               0          |\n",
      "| Dest.missing(NA)          |               0          |\n",
      "| Origin.BIL                |               0          |\n",
      "| Dest.EYW                  |               0          |\n",
      "| Origin.PHX                |               0          |\n",
      "| Dest.CHA                  |               0          |\n",
      "| Dest.TOL                  |               0          |\n",
      "| Dest.PWM                  |               0          |\n",
      "| Origin.MAF                |               0          |\n",
      "| Origin.SDF                |               0          |\n",
      "| DayOfWeek.missing(NA)     |               0          |\n",
      "| Origin.SBN                |               0          |\n",
      "| Origin.HRL                |               0          |\n",
      "| Origin.BNA                |               0          |\n",
      "| Origin.RSW                |               0          |\n",
      "| Origin.STT                |               0          |\n",
      "| Origin.ANC                |               0          |\n",
      "| Dest.SJU                  |               0          |\n",
      "| Origin.ABQ                |               0          |\n",
      "| Origin.AVP                |               0          |\n",
      "| Dest.SCK                  |               0          |\n"
     ]
    }
   ],
   "source": [
    "# Calculate magnitude of normalized GLM coefficients\n",
    "glm_varimp = data_glm.coef_norm()\n",
    "for k,v in glm_varimp.items():\n",
    "    glm_varimp[k] = abs(glm_varimp[k])\n",
    "    \n",
    "# Sort in descending order by magnitude\n",
    "glm_sorted = sorted(glm_varimp.items(), key = operator.itemgetter(1), reverse = True)\n",
    "table = tabulate(glm_sorted, headers = [\"Predictor\", \"Normalized Coefficient\"], tablefmt = \"orgtbl\")\n",
    "print(\"Coefficient Magnitudes:\\n\\n\" + table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.axis.XTick at 0x10e9526a0>,\n",
       "  <matplotlib.axis.XTick at 0x10e7a9fd0>,\n",
       "  <matplotlib.axis.XTick at 0x10e96b160>,\n",
       "  <matplotlib.axis.XTick at 0x10e9c4b38>,\n",
       "  <matplotlib.axis.XTick at 0x10e9cb588>,\n",
       "  <matplotlib.axis.XTick at 0x10e9cbf98>,\n",
       "  <matplotlib.axis.XTick at 0x10e9cd978>,\n",
       "  <matplotlib.axis.XTick at 0x10e9d13c8>,\n",
       "  <matplotlib.axis.XTick at 0x10e9d1dd8>],\n",
       " <a list of 9 Text xticklabel objects>)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7IAAAFKCAYAAAAg3mOpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4LFV97//3BwjGAUHAgKCAUQzGMQ5EDeoRjKJGMbnm\nIiaoaJRfImqu8SdRoxynzHpNNA4M4nVATMBcNYqiwNag4gxoBEFlFlBmgiII3/vHqs1p+vSeztC7\n6uz363n62burVlWvrm9Vd39rrVqVqkKSJEmSpKHYbLkrIEmSJEnSUpjISpIkSZIGxURWkiRJkjQo\nJrKSJEmSpEExkZUkSZIkDYqJrCRJkiRpUExkJUkrQpKXJfmvJD9LcmuSl43MOyDJt5Jc1817Wzf9\n/CQ/Wo/XfH63vuduiPeg9ZPk8V08Xr/M9Viv/UqSZCIrSb2RZPckb0vyzSRXJrmp+3takn9I8rAJ\nyxy22B/mSZ7Xlb01ycw85XYdKXfLOryPOyX58yQnJbk8yS+SXJ3kq0nenOTeS13n+krybODtwM+B\n/w2sBk7r5j0K+BBwF+Bd3bzPdItW91hX67v8Oksyk+TWdVhuKvvJMlorJuu6rdazDpKk9bDFcldA\nktQSUuB1QIBvAccCVwFbAQ8GDgFekeSQqnr32OJL/VF8M/DYJLtX1bkT5r+oW+cvWeL3RJcUHgfc\nA7gY+BTwY+BOwEOBVwKvTPKoqjp9ifVeH0+jvaenVdXlE+YBHFhVXx2bt/d6vu7HgK8Al67netbF\n+ibRG20/WUZfBe4PXDE2fdlOOEiS1s1QvngkaZPVJbGHARcAB1TVaRPKbA/8ObD1pFUs8SX/A3gm\n8CfAoWOvsxnwfODrwM7ATotdaZLfAD5LS1oPBd5WVbeOlbkn8PfAXZdY5/W1E8CEJBba+4QJyWZV\nnbc+L1pV1wPXr886ltFG2U+WU1XdCJyz3PWQJK0/uxZL0jLqutm+FvgF8JRJSSxAVV1RVX9FSwLX\nRwH/RetW+7wkm4/N/z1aa+oR67Dud9K65/5tVf3jeBILUFUXV9VzaK2Ut0ly3yQfSHJx1xX5kiT/\nJ8l9J71Qks2T/FmSryS5NskN3TWuL0mSkXKHdV1Gn9CerukKO9uFlpaQBTh/ZN4u3fJzXsuYZP+u\n+/SVSX6e5LwkxyR5+EiZ2W66a10jm2TnJO9M8sMkNya5IsnHkzxiQtnV3Xoel+RZXTftG7rX/kiS\nnUbK7tq9r8eNvedbk5w86b1MsN77Sdr1wcd17+9nXZxOTfJH8yzzyCQnpl2rfG2SzyV51Oj7Hyt/\na5KTk2yX5PAkP+625XeTPH/C+m93jexit9V82y7J+7v5u0yYd0hXl593+/Y7ksx7Eifteu1T0rrj\n/zzJ95K8NsmWE8o+Nsknk1zUve9Lu2NiWa8BlqRpsEVWkpbXC2ifxcdW1dkLFZ6UHC7RbJJ3BPA+\nYD9a99dZLwL+G/gI7VrRxa002Q3Yh3YN6j8sVL6qbh5Z9pHA54E7A58AvgfsAfwxsF+SfarqmyPl\nt6C1Fj4JOBv4MHAjLVl9B7An8Lyu+Cm0pOwgYJfuPc1ug293z3+f1n37n4Brunmzfyd2N03yfuC5\nwE+B47u/9+zqcDbwzZHia60j7XrnE4FtaK3YxwPb01pAT03yzKr6zNg6CngJ8PRuO80Avw3sDzw4\nyUO77XpN974mvefzJ72fSW+x+7s++8m7gO8CX6C1dm8HPBX4YJL7VdVht3vBlqR+lnaS/XjgR8CD\naDE8mbm7/m4DfIl2MujfgDsAfwi8L8ktVfXBed7nhthWE7slJ/kn4KW0rvXvpXXV3o8Wsy27+o4v\n8z7aiZWLaF30rwEeBbwJ2DvJ785+BiTZl3YcXEvbHy4BtqV1nf5T4I2LrL8kDVNV+fDhw4ePZXoA\nJwG3AAet4/KHAbcCr19E2ed1Zd9I6/57DXDCyPydaD+239M9vwi4ZZH1+ONu3V9ch/dwVrcNnj02\n/Q+7dX5vbPrqbvrbgYxMD3Bkt66njy1zylzvBTi6W2aXCfPOA340Nu3F3et/BbjL2LwAO4xt81uA\n545M2xz4AfAzYK+x5XekXVt8CfArE+J8DfCbY8t8uHuNZy32PU9jPwHuPWHaFrSTFr8A7jG23c7t\n3seT5tjetwCPG5s3O/29Y/vC/bs6fnes/OMnHS8LbatumZMXu/8Aj+6W+T6w9cj0LYEvd/PG96vn\nd9P/DdhybN7ru9d46ci047tpD5xQp22XGncfPnz4GNrDrsWStLx27P5eMj6j6/a4uuseO/t4+YZ4\n0ar6GXAM8LsjXSJfSGsNW5duxffo/l68lIWSPAb4DeDLVXXsWB3/DTgV+I0ke3XlQxv46lLgFVVV\nI+UL+Ivu6ZzdVzeAl9Ja4A6uqv8eq3PV5OtwRz0N+HXgHVV16tjyl9G6j+9Ia+Ee909V9b2xaUfQ\nEsE9F/8WFmd99pOacH1xVf0S+BdaQjv6/h4D3IeWLJ44ttgRzH9d68+AvxjbF86itdLeP8md5qvn\nRvIC2j7ylqq6dqReNwGvnmOZl9OS7xd25Ua9mTb42+h+Pft+bxxfUVVdtY71lqTBsGuxJPXXbrSW\nmNkfrKF1d/ynDbT+I4D/D3hhktW0H99n1kg33imYvaXQKXPMPxn4HeC3aEnt/WjdJ88BXpesNc5V\naN2b77/Ba0q7tRDwAOCyqjpzHVfz6O7vbmkDfY3bnfY+7s+a2wBB2w8mxeai7u/d1rE+C1mn/STJ\nvYC/pI38vAtwx5HZxZpBtqDFF1ryeTtVVUm+TNsuk5w7fkKhM7pdfjZfXTeC2ffzxQnzTqW1pN4m\nyR1p3dt/CvyvOfbrX3D7/frDtG7xX0vyUdox9KWqWuukmCRtikxkJWl5XUa7HnStUV+r6gt0g/Kl\njRL7yw35wlX17STfol0f+FVgV9Z9MKnZEX93nrfU2ramJTVz3Z7mUtqP+G2659t1f3enJflzufMS\n67FYs/VYn2Rh9j08a54yRRs4a9w1E6bN7hfjAzJtEOuyn6QNYvZ1Wnz/k3bt67W0BG43WvflO4ws\nMrsfzNWaPV8r96RtAht5uyxgdnTxtepdVbckGb/9z91o+/ndmX+/Hm11/vckv0frhXAQrQt2knwT\neHVVfX496i9JvWfXYklaXl+i/YCd1I101FJvsbNYh9MGKXoPrdXqw+u4ntkuso9IstUSlruW9t52\nnGP+PWg/3q8dKQ/w71W1+TyPiaMdbwCzSdNSE/ZR19Le0zMWeA9vWv/qbjBL3U/+gpacvaCq9q6q\nP6+qw6rqjbRBrsb35+u6aTvMsb65pk9LMffJ/20mTJvdT9eqdzcC9PZzlP/2AvvE7epQVSdU1RNp\n23of4G20HgOfTLLH4t6aJA2TiawkLa/301qOnpV2H9ZpOwa4gZaY/WtVXbcuK6mq82mD+Pwq8P8v\nVH7kViLf7v6umqPo3t3fb3V/z6YbyXXCLWE2uu6a0e8COyR5yDqu5jRa0va4hQqup1vgtuuK19dS\n95P7dH8/NmHeKtYe5Xd2P9hrvHBX/8csuqbrZqFtdTVwr/GJXU+Jh04oP7u/Pn7CvMcy1kpcVTfQ\nbnf0gCSTEuN5VdXPq2qmql4J/DVtUKmnLHU9kjQkJrKStIyq6ke0gVzuAHwmyaPnKDrf9Y9z3ZZk\nMa//38CTadfavW5d19N5GXA98Ookr5iUaCbZJcmxtFuKUFVfoo3suleS/zFW9lm0xOb7s4MiVdUt\ntFvs7AS8I8mvTniNHZNslGtkO/9MS0TfO35P0DRztS7P+jjwQ+AlSSYmG929U9d6b0t0Zfd3rfub\nLtU67Cfnd39XjU5M8mTaYFHjvkTbJk/obisz6mDatdEb00Lb6mvALkmeODb9dbSu1uPeT9tHXpvk\ntmO3i+nfzPEab6N9DhydZOvxmUm2SfJbI88fO8fJnNn9b9rXBUvSVHmNrCQts6p6Y9cQ9DrgS901\nbl+jjVK6De2awifSEtYvTFhFgN/vrkuc5MSq+sg8r//lda/97dZzdpIn0e5/+Q/Ay5OcRLuP5p2B\nh9AGbrqV2/+Yfx6tu+lHk3yc1uq6B+2em9fS7tc66k20gXEOBp6e5GTaNau/Rrt29neA19Bu67PB\nVdWR3SjKBwLndnX+KS253hs4itvfwzNjy/8yyR/QBnL6VDeQ0em0xONewCOBe9O6Va81Iu0SnES7\nhdG/J/k0bRCsC6rqQ+uysiXuJ++iXbd5XJLjaPvAA2nJ8L8Czx5bdyX5E+AE4BNJjqcltg+m7fuf\nprUwru99lOey0Lb6x67un+gGVrqK1kq8G+1+vrdrea2qLyd5B22E7e9222D2PrJXMeGa8Ko6uru/\n8J8BP0zyWeBC2uBm96a14L+vmw/thMrOSb5EO3FwE/Bw2j54HnC7UcAlaVNjIitJPdAlsx+hjQ77\nBOAAWvJ3Pe0H/b8AH6qq0yctTvvB/+A5Vn818JGRsktpwV1Sa29Vfa27Nu9FwDOAp7Jm1Ngf0BLc\nI6rqgrFlHgn8FS1p+T3gCtp1mG+uqnPHXuOXtMT9j2j33nwabWCkn9J+wL+WyddwzvdeljSvqp7f\nJRovpiVAd6AlJ18APrGI5b/TdU1+Be39Pp+WpF1K65b6Oto2WKxJcT2S1sL4bFp37y26+i0mkV2v\n/aR7f6tovQ2e2r32GbQW3euA/Scs84Ukjx9ZBlo37CfQ7lNMt+xS6jlp3pK3VVWdnGQ/2kBM+9O6\nWZ8I/E/aSYtJMX55ku8DL6HtJ1fSulq/FjhzjmVemuQE2ufAPrQTWVfREtq/4/b79Vto2/MRXdlb\nu3Jvpt2m6VokaROWkduubfwXS46ifWFfXlVr/eBKsh3tS+MetOtH3lpV759aBSVJUu90rY6PBLau\nqp8vd30kSctv2tfIHk3rmjOXQ4DTq+qhtDOwb01iq7EkSZu4JHec49rQ59PuvftZk1hJ0qypJolV\ndWqSSYMizLoMeFD3/1bAlV0XMkmStGnbBfh2ks/RuqFvAfwWbcCvq4BXLmPdJEk907fWziOAk5L8\nmHa90/7LXB9JkjQdl9MuL3o8bbTjO9BOcB8F/HVVnbd8VZMk9U3fEtlXA2dU1ROS3Af4XJIHd8P+\nS5KkTVRVXUMbFEmSpAX1LZH9HdoofFTVD5OcR7sFwzfGCyaZ3ihVkiRJkqSpq6pMmj7twZ6g3U9v\nYmVo9/x7IkCSHWg3QP/RXCuqql4+DjvssGWvgw9j0beHsejXw3j052Es+vMwFv15GIv+PIxFvx4r\nLR7zmWqLbJJjaNe9bJfkQuAwYEvavdAPB/4GODrJGbRk91VVddU06yhJkiRJ6rdpj1r8nAXmXwE8\nfUrVkSRJkiQN0HJ0Ld7krVq1armroI6x6A9j0S/Goz+MRX8Yi/4wFv1hLPrFeKyRhfoe91WSGmrd\nJUmSJEnzS0L1aLAnSZIkSZLWmYmsJEmSJGlQTGQlSZIkSYNiIitJkiRJGhQTWUmSJEnSoJjISpIk\nSZIGxURWkiRJkjQoJrKSJEmSpEExkZUkSZIkDYqJrCRJkiRpUExkJUmSJEmDYiIrSZIkSRoUE1lJ\nkiRJ0qBssdwVWB9JlrsKU7PDrrty2fnnL3c1JEmSJGnZpaqWuw7rJEkx0Lqvk4ShxkqSJEmSliot\nB5rYemnXYkmSJEnSoJjISpIkSZIGxURWkiRJkjQoJrKSJEmSpEExkZUkSZIkDcpUE9kkRyW5PMmZ\n85RZleTbSb6b5JRp1k+SJEmS1H9Tvf1Okr2A/wY+UFUPnjB/a+DLwJOq6pIk21fVFXOsy9vvSJIk\nSdImqje336mqU4Gr5ynyHOD4qrqkKz8xiZUkSZIkrVx9u0b2fsC2SU5J8vUkBy53hSRJkiRJ/bLF\ncldgzBbAw4C9gTsDX0nylar6wfJWS5IkSZLUF31LZC8GrqiqG4Ebk3wReAgwOZFdvXrN/6tWtYck\nSZIkaXBmZmaYmZlZVNmpDvYEkGQ34JNV9aAJ8/YA3gHsC9wB+Cqwf1V9b0JZB3uSJEmSpE3UfIM9\nTbVFNskxwCpguyQXAocBWwJVVYdX1dlJPgucCdwCHD4piZUkSZIkrVxTb5HdUGyRlSRJkqRNV29u\nvyNJkiRJ0vrq22BPGqAdd9uNyy+4YLmrMTU77Lorl51//nJXQ5IkSVqx7Fo8FD3uWpwEYyFJkiRp\nQ+rNYE+SNr6V1EJu67gkSdLKZIvsUPS4FdAW2X5ZUfHoeSwkSZK07hzsSZIkSZK0yTCRlSRJkiQN\niomsJEmSJGlQTGQlSZIkSYNiIitJkiRJGhQTWUmSJEnSoHgfWUnaSLynryRJ0sbhfWSHosf3y1xR\n9y2FXscCVlg8jEV/9DwWkiRpeLyPrCRJkiRpk2EiK0mSJEkaFBNZSZIkSdKgONiTJGmTt5IG3gIH\n35Ikbfoc7GkoejyQyooa0AZ6HQtYYfEwFv1hLPql5/GQJGkxHOxJkiRJkrTJMJGVJEmSJA2Kiawk\nSZIkaVBMZCVJkiRJgzLVRDbJUUkuT3LmAuUemeTmJH8wrbpJkiRJkoZh2i2yRwNPnq9Aks2AvwU+\nO5UaSZIkSZIGZaqJbFWdCly9QLGXAscBP9n4NZIkSZIkDU2vrpFNshPwzKp6NzDxfkGSJEmSpJWt\nV4ks8Hbg0JHnJrOSJEmSpNvZYrkrMOYRwLFJAmwPPCXJzVX1iYmlV69e8/+qVe0hSZIkSRqcmZkZ\nZmZmFlU2VbVxazP+gsluwCer6kELlDu6K/exOeYXU677skqYdqwWKwnGoj9WVDyMRX8Yi37peTwk\nSVqMtO+zib10p9oim+QYYBWwXZILgcOALYGqqsPHivsNLEmSJElay9RbZDcUW2T7w5aOfllR8TAW\n/WEs+qXn8ZAkaTHma5Ht22BPkiRJkiTNy0RWkiRJkjQoJrKSJEmSpEExkZUkSZIkDYqJrCRJkiRp\nUExkJUmSJEmDYiIrSZIkSRoUE1lJkiRJ0qCYyEqSJEmSBsVEVpIkSZI0KCaykiRJkqRBMZGVJEmS\nJA2KiawkSZIkaVBMZCVJkiRJg2IiK0mSJEkaFBNZSZIkSdKgmMhKkiRJkgbFRFaSJEmSNChbLHcF\nJEnSyrHjbrtx+QUXLHc1pmaHXXflsvPPX+5qSNImJ1W13HVYJ0mKgdZ9nST0NVZJMBb9saLiYSz6\nw1j0S4/jYSwkSYuV9hmaSfPsWixJkiRJGhQTWUmSJEnSoEw1kU1yVJLLk5w5x/znJDmje5ya5EHT\nrJ8kSZIkqf+m3SJ7NPDkeeb/CHhcVT0EeDNwxFRqJUmSJEkajKmOWlxVpybZdZ75p408PQ3YeePX\nSpIkSZI0JH2+RvZPgBOWuxKSJEmSpH7p5X1kkzwBOAjYa7nrIkmSJEnql94lskkeDBwO7FtVV89b\nePXqNf+vWtUekiRJWtCOu+3G5RdcsNzVmJoddt2Vy84/f7mrIWkeMzMzzMzMLKpspn2T7iS7AZ+s\nqrVGJE6yC3AScODY9bKT1lPeUL0fvLl9v6yoeBiL/jAW/dLjeBiL/jAWkvou7bjNpHlTbZFNcgyw\nCtguyYXAYcCWQFXV4cDrgG2BdyUJcHNV7TnNOkqSJEmS+m3qLbIbii2y/eEZ3X5ZUfEwFv1hLPql\nx/EwFv1hLCT13Xwtsn0etViSJEmSpLWYyEqSJEmSBqV3oxZLkiRJK81KGkXaEaS1IXiN7FD0+LoO\nr7HplxUVD2PRH8aiX3ocD2PRH8aiX1ZUPHoeC/WH18hKkiRJkjYZJrKSJEmSpEHxGllJkiRJ6ni9\n8jB4jexQ9PhaghV1TQf0OhawwuJhLPrDWPRLj+NhLPrDWPTLioqHseiPAcTCa2QlSZIkSZsEE1lJ\nkiRJ0qCYyEqSJEmSBsVEVpIkSZI0KCaykiRJkqRBMZGVJEmSJA2KiawkSZIkaVBMZCVJkiRJg2Ii\nK0mSJEkaFBNZSZIkSdKgmMhKkiRJkgbFRFaSJEmSNCgmspIkSZKkQTGRlSRJkiQNylQT2SRHJbk8\nyZnzlPnnJOcmOT3JQ6dZP0mSJElS/027RfZo4MlzzUzyFOA+VbU7cDDwnmlVTJIkSZI0DFNNZKvq\nVODqeYrsB3ygK/tVYOskO0yjbpIkSZKkYejbNbI7AxeNPL+kmyZJkiRJEtC/RFaSJEmSpHltsdwV\nGHMJcK+R5/fspk22evWa/1etag9JkiRJ0uDMzMwwMzOzqLKpqo1bm/EXTHYDPllVD5ow76nAS6rq\naUkeBby9qh41x3qKKdd9WSVMO1aLlQRj0R8rKh7Goj+MRb/0OB7Goj+MRb+sqHgYi/4YQCyqKpPm\nTbVFNskxwCpguyQXAocBWwJVVYdX1aeTPDXJD4AbgIOmWT9JkiRJUv9NvUV2Q7FFtj9W1Fkr6HUs\nYIXFw1j0h7Holx7Hw1j0h7HolxUVD2PRHwOIxVwtsg72JEmSJEkaFBNZSZIkSdKgmMhKkiRJkgbF\nRFaSJEmSNCgmspIkSZKkQTGRlSRJkiQNiomsJEmSJGlQTGQlSZIkSYNiIitJkiRJGhQTWUmSJEnS\noJjISpIkSZIGxURWkiRJkjQoJrKSJEmSpEExkZUkSZIkDYqJrCRJkiRpUExkJUmSJEmDYiIrSZIk\nSRoUE1lJkiRJ0qCYyEqSJEmSBsVEVpIkSZI0KCaykiRJkqRBMZGVJEmSJA3K1BPZJPsmOTvJOUkO\nnTB/uyQnJDk9yXeSPH/adZQkSZIk9ddUE9kkmwHvBJ4MPAA4IMkeY8UOAU6vqocCTwDemmSLadZT\nkiRJktRf026R3RM4t6ouqKqbgWOB/cbKXAZs1f2/FXBlVf1yinWUJEmSJPXYtFs6dwYuGnl+MS25\nHXUEcFKSHwN3AfafUt0kSZIkSQPQx8GeXg2cUVU7Ab8F/EuSuyxznSRJkiRJPTHtFtlLgF1Gnt+z\nmzbqd4C3AFTVD5OcB+wBfGOtta1eveb/VavaQ5IkSZI0ODMzM8zMzCyqbKpq49Zm9MWSzYHvA/sA\nlwJfAw6oqrNGyrwVuK6q3pBkB1oC+5CqumpsXcUU677sEqYZq6VIgrHojxUVD2PRH8aiX3ocD2PR\nH8aiX1ZUPIxFfwwgFlWVSfOm2iJbVbckOQQ4kdat+aiqOivJwW12HQ78DXB0kjOAAK8aT2IlSZIk\nSSvXVFtkNyRbZPtjRZ21gl7HAlZYPIxFfxiLfulxPIxFfxiLfllR8TAW/TGAWMzVItvHwZ4kSZIk\nSZqTiawkSZIkaVBMZCVJkiRJg2IiK0mSJEkaFBNZSZIkSdKgmMhKkiRJkgbFRFaSJEmSNCgmspIk\nSZKkQTGRlSRJkiQNiomsJEmSJGlQTGQlSZIkSYNiIitJkiRJGhQTWUmSJEnSoJjISpIkSZIGxURW\nkiRJkjQoJrKSJEmSpEExkZUkSZIkDYqJrCRJkiRpUExkJUmSJEmDYiIrSZIkSRoUE1lJkiRJ0qCY\nyEqSJEmSBmXqiWySfZOcneScJIfOUWZVkm8n+W6SU6ZdR0mSJElSf20xzRdLshnwTmAf4MfA15N8\nvKrOHimzNfAvwJOq6pIk20+zjpIkSZKkfpt2i+yewLlVdUFV3QwcC+w3VuY5wPFVdQlAVV0x5TpK\nkiRJknps2onszsBFI88v7qaNuh+wbZJTknw9yYFTq50kSZIkqfem2rV4kbYAHgbsDdwZ+EqSr1TV\nD5a3WpIkSZKkPph2InsJsMvI83t200ZdDFxRVTcCNyb5IvAQYO1EdvXqNf+vWtUekiRJkqTBmZmZ\nYWZmZlFlU1UbtzajL5ZsDnyfNtjTpcDXgAOq6qyRMnsA7wD2Be4AfBXYv6q+N7auYop1X3YJ04zV\nUiTBWPTHioqHsegPY9EvPY6HsegPY9EvKyoexqI/BhCLqsqkeVNtka2qW5IcApxIuz73qKo6K8nB\nbXYdXlVnJ/kscCZwC3D4eBIrSZIkSVq5ptoiuyHZItsfK+qsFfQ6FrDC4mEs+sNY9EuP42Es+sNY\n9MuKioex6I8BxGKuFtlpj1osSZIkSdJ6MZGVJEmSJA2KiawkSZIkaVBMZCVJkiRJg2IiK0mSJEka\nFBNZSZIkSdKgmMhKkiRJkgbFRFaSJEmSNCgmspIkSZKkQTGRlSRJkiQNiomsJEmSJGlQTGQlSZIk\nSYNiIitJkiRJGhQTWUmSJEnSoJjISpIkSZIGxURWkiRJkjQoJrKSJEmSpEExkZUkSZIkDYqJrCRJ\nkiRpUExkJUmSJEmDYiIrSZIkSRoUE1lJkiRJ0qBMPZFNsm+Ss5Ock+TQeco9MsnNSf5gmvWTJEmS\nJPXbVBPZJJsB7wSeDDwAOCDJHnOU+1vgs9OsnyRJkiSp/6bdIrsncG5VXVBVNwPHAvtNKPdS4Djg\nJ9OsnCRJkiSp/6adyO4MXDTy/OJu2m2S7AQ8s6reDWSKdZMkSZIkDUAfB3t6OzB67azJrCRJkiTp\nNltM+fUuAXYZeX7PbtqoRwDHJgmwPfCUJDdX1SfWWtvq1Wv+X7WqPSRJkiRJgzMzM8PMzMyiyqaq\nNm5tRl8s2Rz4PrAPcCnwNeCAqjprjvJHA5+sqo9NmFdMse7LLmGasVqKJBiL/lhR8TAW/WEs+qXH\n8TAW/WEs+mVFxcNY9McAYlFVE3voTrVFtqpuSXIIcCKtW/NRVXVWkoPb7Dp8fJFp1k+SJEmS1H9T\nbZHdkGyR7Y8VddYKeh0LWGHxMBb9YSz6pcfxMBb9YSz6ZUXFw1j0xwBiMVeLbB8He5IkSZIkaU4m\nspIkSZKkQTGRlSRJkiQNiomsJEmSJGlQTGQlSZIkSYNiIitJkiRJGhQTWUmSJEnSoJjISpIkSZIG\nxURWkiRJkjQoJrKSJEmSpEExkZUkSZIkDYqJrCRJkiRpUExkJUmSJEmDYiIrSZIkSRoUE1lJkiRJ\n0qCYyEqSJEmSBsVEVpIkSZI0KCaykiRJkqRBMZGVJEmSJA2KiawkSZIkaVBMZCVJkiRJg2IiK0mS\nJEkalKknskn2TXJ2knOSHDph/nOSnNE9Tk3yoGnXUZIkSZLUX1NNZJNsBrwTeDLwAOCAJHuMFfsR\n8LiqegichrioAAATkklEQVTwZuCIadZRkiRJktRv026R3RM4t6ouqKqbgWOB/UYLVNVpVXVt9/Q0\nYOcp11GSJEmS1GPTTmR3Bi4aeX4x8yeqfwKcsFFrJEmSJEkalC2WuwJzSfIE4CBgr+WuiyRJkiSp\nP6adyF4C7DLy/J7dtNtJ8mDgcGDfqrp6zrWtXr3m/1Wr2kOSJEmSNDgzMzPMzMwsqmyqauPWZvTF\nks2B7wP7AJcCXwMOqKqzRsrsApwEHFhVp82zrmKKdV92CdOM1VIkwVj0x4qKh7HoD2PRLz2Oh7Ho\nD2PRLysqHsaiPwYQi6rKpHlTbZGtqluSHAKcSLs+96iqOivJwW12HQ68DtgWeFeSADdX1Z7TrKck\nSZIkqb+m2iK7Idki2x8r6qwV9DoWsMLiYSz6w1j0S4/jYSz6w1j0y4qKh7HojwHEYq4W2WmPWixJ\nkiRJ0noxkZUkSZIkDYqJrCRJkiRpUExkJUmSJEmDYiIrSZIkSRoUE1lJkiRJ0qCYyEqSJEmSBsVE\nVpIkSZI0KCaykiRJkqRBMZGVJEmSJA2KiawkSZIkaVBMZCVJkiRJg2IiK0mSJEkaFBNZSZIkSdKg\nmMhKkiRJkgbFRFaSJEmSNCgmspIkSZKkQTGRlSRJkiQNiomsJEmSJGlQTGQlSZIkSYNiIitJkiRJ\nGhQTWUmSJEnSoEw9kU2yb5Kzk5yT5NA5yvxzknOTnJ7kodOuoyRJkiSpv6aayCbZDHgn8GTgAcAB\nSfYYK/MU4D5VtTtwMPCeadZxg5iZWe4aaJax6A9j0S/Goz+MRX8Yi/4wFv1hLPrFeNxm2i2yewLn\nVtUFVXUzcCyw31iZ/YAPAFTVV4Gtk+ww3WquJ3ew/jAW/WEs+sV49Iex6A9j0R/Goj+MRb8Yj9tM\nO5HdGbho5PnF3bT5ylwyoYwkSZIkaYVysCdJkiRJ0qCkqqb3YsmjgNVVtW/3/C+Bqqq/GynzHuCU\nqvpo9/xs4PFVdfnYuqZXcUmSJEnS1FVVJk3fYsr1+Dpw3yS7ApcCzwYOGCvzCeAlwEe7xPea8SQW\n5n5DkiRJkqRN21QT2aq6JckhwIm0bs1HVdVZSQ5us+vwqvp0kqcm+QFwA3DQNOsoSZIkSeq3qXYt\nliRJkiRpfW3UwZ6S3CvJdUl23BjltWEYJ2lpPGaklSHJ9Ul+e7nrsbEk+W6SP1zuekgbi9/Xm7ZF\nJ7JJHp3khCTXdB/sX0/y3PmWqaqLququVXXZYl5jqeXnqOetSW4a3wGTHNrNe9/ItPOT/DzJtUmu\nTnJWkvck2X2kzFuTfH5sXW/t1vUbI9N266Y9cF3rviEMLE6PmW96ksd3z6/rHhcneV+Sbbr5z+vm\nHzm2jtcmOWVd67ahJTkyyclj05Lki0nePaU6PCPJKUmu6B4nd9egj5b5ja7Mfye5oLsMYHT+XZJ8\noNu3rkzy7iS/MjL/FUnOGZl/SpLHTuP9rY+hHzPdvPOSPKf7f9eu7E5jZSZOXy4r6Lg4qduvZj/H\nbujisO803uOGkmQmyY0j35c/SPLBJA/fQOu/bR+ep8xhST43Yfre3TbefWz6FUl+mmTrJG9KcsL4\na41On6SqturuaT81Se7bbdsfd/vMD7rj5T4b+rWq6oFV9W8bcp1zbdPx6UlO7fap67p96htJnrkh\n67Ix9fyYOCzJid3/o58/N3WP62and2V+Pcm/Jrm0m3dBkuOTTHssnXn1/fs6yV5j2/uWtDxj9vmn\nunKL+j5fzPSVblGJbJInAScDXwLuDdwd+Hvg7UkOm2OZX5k0fUrOYe1ra18IfG9sWgEvqKqtq+pu\nwDOAAKcn2bMrcxLw6CRbjiy3N/Dd7u+sfYDLq+q7G+g9LNkA47QYv+w+UO4K7AU8Gnj7yPzrgOdk\n7RMIfeoz/3Lgnkn+YmTaocD2wP/akC+UZvMJs7YG3krbL34N+Djw2SS/1i33K8CngK8B2wLPAt6Q\n5Okj63g3cK9uHb8JPBz425H5/xf47araBtiRdi38f8xRn17YRI8ZmHv/97i4vY1+XFTVPl0yNPs5\ndihwGbDWj8+eK+CNI9+XTwDOB76SZL8p1+P2E6pOpsXhw0k2A0hyP1rM3l1V18617ALTpy7Jg4Bv\nAL8AHt3tM48Avgk8dR3XudZn1ob4HEuyeZK5Bt5czLYu4PXde9weOA741yT3Xt+6TUlvj4nbzbz9\n58//AT7UPd+qmwbtc+4SYPdu2qOBz9J+E/fCEL6vq+rUse39I+DFs8+r6mnTrM+KUFULPmiJ4ZET\npj8PuAnYBTga+FD390rgX4BdgVuBnUaWeQ1wEXAF8Dbg87QPMsbLA4d1898CXE778l+9QF1vBf4c\n+MHItFXAd4DDgfeNTD8PeM6EdXwO+M/u/zvTvlBWdc+3A64FDgSOH1nmGNqHw6K26cZ4DDBOj5lv\nOvB44Kax+f8AnDHyvs6hJbafGSnzWuDk5YzFhPe1Jy3pfiDwUOB64KEj+9T7uu19WbcvbT+y7J8D\nZ3fLnwe8aWTe5t02O4T2Q+cG4GGLrNPVwL7d/08BrgG2HJn/98Cnuv+36o6DR43Mf2q3zGYT1n0H\n4FXA2cu97Tf1Y6abd9tnWfdat4zWbb7pHhfTOy66+d8ZretQHsApwGsmTD8SuLD7/47AP9J+vF0B\nfBq4z0jZZ9NOKF9Lu3PB0d30T3T75s+6eH5mjjocBpw4x7xfAb4FvKGL/2ldHGaPizcBnx49XmjH\n+ZXAf3X72ZXAe+jGD+nK3u64A14A/KBb9weAD9L9rmDy58LzgHNHni+0jT4PfH6BWOzdvb+raJ8f\nHwHuPhar/w38e1fPV83WA3hl916/M7otRpZ9IPAZ4Ce0pOyvgc3H3t8Lum32c+DXJtTvtm0933Tg\nP4FXjTzfulv/05d7fx/6MTHP9CMY+R3cTdu22+57LPc2XWB7D+b7euR1zgWeO2H6or7PFzN9pT8W\nbJHtuurcF/jwhNnH0Fp1f7d7/izaWZ3tgdmz7DWyrucCLwWeBuxAO2gfN7bOGnv+WNqH6T2A/YDX\nJHn0AtX+GnB9kn265y+iHbyLPbP0UeBRSX61qm7o1je7rr2BU2nJ7qqRZfam7ejLYqBxWpIk9wWe\nTovHqDcCv92dreulqvoa8He0WHyQdhb39G72J4EbgT2A3br/PzSy+IXAk6qd3ft94OAk4z0ODgL+\nALgL7YfyvLquT3em9SwAeDDwvaq6aaTYt4CHdP8/gDbK+bfG5m8F/PrIevdOcjXty/fFtHj10ko4\nZsb05sz6rJVyXIysf6/u/RyxUF0G5Fhg564F9EjgfrQTFDsCX6XrlZHkjrTE70+ramva9jkSoKqe\nQYvnC6u1Wiy523VV3Qz8Ea01/xhaXK5exKLbAFt29dkT+ENacrGWtEsl3kn7bNuW9jtg//GqTKre\nyP8LbaPHd/Wfz4202xRuBzyI9hny9rEyBwFvr9ZD5p+7abt1r3lf4JET3t/dgRlay+g9aK1yTwRe\nPVb0ANrvn62Any5Q10Xper0dQjsxdOaGWOcy6sUxsVhVdRXtM+/IJAcmuf/Geq11tQK/r7VIi+la\nfHdaQC8Zn9F9cVxB65IFcGpVHVfNjRPWdSDw3qo6s6puqap/AH68wOufU1VHVNWt1a5TOZ3WzWYh\nRwIvTrIt7Qz5BxexzKyLadvmbt3zz7Mmkd2H1tp3GXBZkod13VrvzjImsgwzTickuWrkcTVrf3hs\n0c27ktbN5STWfDDNvr+rgL+htZT02d/QfoBc321T0gYReSDwsqq6oYvHq4EnzXZvrKqPVdWF3f+n\n0z7I9xlb999X1QVdTG+erxJp10j+K61V6OJu8la0M8KjrgFmux3dhdY6ftPY/IyUoapOrtbFajvW\ndC3ua1fcIR4zixXgv0aPL+CMDbTuDW2TPy5GvJjWSnLhfHUZmNlttSMtwfmzqrqiqn5Ja4G7BzA7\nWNJNwP2T3K2qfl5VXxpb13qdbKmqs4DVtJMXk64le2J3LOxC66H1Ltp3/blVdXNV/ZD2HTPXcXgg\n8G/d59ytVfVB1j6xOqck2zP/NrobrTV5rc+ksff55ar6Zrdf/4TWU2l83z+uqr7QlZ/9zLoJ+Muq\n+sUcn2PPBU6vqiO7z7FLad3knzdWbnVV/bSqfllVkxJ36Lb12Pf7X0wod1gXkwuBfYHfr6oL5nv/\nAzDtY2LVhG196BLrvIp2EuPlwLeTXJbktUtcx8a0KX9fT/LeCTG910Z8vcFaTCL7U9qBtPP4jO4H\n6vasOSN3/gLr2hkY/4C6aIFlLh17fgPtx8VCPgQ8idaN5lNVtZgzs7PuRWv2n13mJOARSbaitbzO\nDlAy0z3fm/ZFeDHLZ4hx2reqth153I21P7R/2c3brqruU1UvqarrJqzrn4C7JnnBAq+5bKrqVlp3\nrNHrqO8N3An4yUjCcTaty9YuAEn+KG1Agyu7D7ODaR/qoxb1xZ/kXrTuUMdX1ZtGZl1P69Y1ahta\nl6bZ+VuOXSu+Tfd3rXhU1TXAy7r30NcBn4Z4zCxWAb85enzRWhd7Z6UcF0nuRmspmMpAVlN0z+7v\nLd3fM0didiWtZfReVfVz2knlpwA/7GJ3wEaozxnAzVV1DnAzrcvxrM93x8KltBbLP6N1zx0133F4\nT9b+LDhvCXXbrfs7cRvRfnPcwoTPpFHdCfTPpA3Mcw1d1+KxYuP1BLi0S6bmcm9gr7ETYO9jTYIA\n7bNlMcfV5yd8v791Qrk3dPN3rKrHVtVnFrHuvpv2MTEzYVv/3VJWUFVXVdVfVdUjaJ9hr6KdZHj+\nOtRnY9iUv68nefGEmC5UxxVpwUS2qs6l9e2fdHbzAFrCNztoxa0LrO4SWt/zURvlDEO1wR0+TjsY\n37vExfcHThs5k3MarcXgucC2VfXtbvrJtG43+7C8rbFDjdMG6+pYVb8A/orWzfhOG2q9U3ABcO34\nB1ZV3bmqvpFkN1rXo9fTroG6G21/Ht92C8V0tmv2F4Fjq+pVY7PPAH5zrPX0Yaxpxfse8Mtu2qyH\n036s/2iOl9y8e1y/UN2Ww0CPmaXoXVfiJdgUj4uDaD+0PrVQnQbm2bT9/1xakrP7WMzuUlUfBaiq\nL1bVfrQeG28BPjQysM+CsVoH59O6I94myV1oLWVzfW7N5xLWJKOzRp9fT9sH7zwybfSH9wXMs426\nxGaG9vkzn2Np137ft1rX4UnlJ23PhbbxBcDnxo67bap1e13Kela6Ph8TC6qqG6vqA7Qu3g9djjqM\nWwHf1+OG/P09VYu9/c4hwB8neU2SuyX51STPog0m8LcLdAMZDcYHad19H5Jki7TRKu8xT/n1dSjw\nu1X1n4spnDbk/btp14W8cnZ6dwbzP2mDCH1hZJEZ4HdofeuXNZHtDDVOG0RVHUPrHvKi5a7LEnwV\nOCvJ27sWG5L8WpL/2c2/S/f3iqqaHa79j5b6Il339y8CR1TVYROKnET7kf2mJHdIG7X7hbSBT6iq\n62k/nt6SZLsk96ANgHBU16JGkj/tps9ea/Vu2uAMp6/1av0xtGNmyy4+s4+5um3P9Vq9O27nsMkc\nFyNe1L3OXF0xByXJPZO8gXaC92XVurgeA7y766ZNkm2SPDPJnbr4/UGSu3bb4Fraj/zZVqvLgN0n\nvNS4zcaOgTskc46c+37acblX93wLWu+dM4Bvz7HMfD4IPCvJE9Kucfxj1nQRnb3M5XzgBUk2SxuB\n+E9G5v+UebZRV+wVtB5gh3cnbEi7fdDBSV7albkr7UTPDUl2Af5yHd7LJB/oXvug2e2adluWJ4+U\nGcpnyNQt4zGxvvXeJslfJ3lA9/21eZL/QRsD4Isb+/WXYGjf1wtZ7Pe55rGoRLbr6rEPbRCC82lf\n7K8GXlFVr19o8ZH1fIA2gtinaQfoTrTWzl9MKr/Q+uC2+2MdMGl+VV1eVacssL4j0+4Bdg3wH7Rt\n8lu19n3jPk+7KPykkfVfCfyQdvb1ZJbZUOM033Lr4JW0M5uD+LHY/dh9Om2wkW91++GX6LrjVrud\n05uAT6d1n3wFaw8EstZ7TfKCJD8ZmfQa2v77l10sZu9z9vvd69xMG/jgUbSudsfRRvD75Mg6XkLr\n2vJDWlfQb3D7H1CPAb6Z5Hpaa8FmtO7j816buJwGeMycRBtI62e0brZnTFp2ntfyuJj+cUGSxwP3\nAY5a6rbomdeNfF9+gTY4zaOr6v92819E6wI+k+Ra2v75LFosNqNtq/O6ee+gjeY5e73wm4ED07qK\nz95r8dVJxgfpegK3PwZ+RhugaS3dyc3X0I7NV9L2nzsAz5hwomEuo8f5F2mDxBxF6yL6JNqJjFHP\no+2719BGqz1ybP5824iq+g5tIKY70m7jch1tALGHs6Y1/0XAi7p5x9Gu7Z5Y5yW+v8tp2/eZtM/D\nq4DjaV2O51x3kr9Ksi4nBgbxebSAPhwT6+smWvfx42n79U9ox81Lq+pjG/i11tnAvq8XWs9Svs8X\nW6cVKct5crg7i3oR8MqqGv8yUE8YJ2lpPGaklSHJEbTb0/R2fAZJc/P7etgW27V4g0myf9eEfifa\n9Yx3BE6Ydj00P+MkLY3HjCRJ/ef39aZj6oksrY/7ZbSLrVcBT+kGZlK/GCdpaTxmJEnqP7+vNxHL\n2rVYkiRJkqSlWo4WWUmSJEmS1pmJrCRJkiRpUExkJUmSJEmDYiIrSZIkSRoUE1lJkiRJ0qCYyEqS\nJEmSBuX/AU0qx464aQCRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e74ca90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot GLM Coefficient Magnitudes\n",
    "all_coefficient_magnitudes = pandas.DataFrame(glm_sorted)\n",
    "coefficient_magnitudes = all_coefficient_magnitudes[1:10]\n",
    "feature_labels = list(coefficient_magnitudes[0])\n",
    "Index = coefficient_magnitudes.index\n",
    "\n",
    "# for python3 use range() instead of xrange()\n",
    "plt.figure(figsize=(16,5))\n",
    "h = plt.bar(range(len(feature_labels)), coefficient_magnitudes[1],width=0.6, label=feature_labels, color ='aqua')\n",
    "plt.title(\"GLM Coefficient Magnitudes\", fontsize=20 )\n",
    "xticks_pos = [0.65*patch.get_width() + patch.get_xy()[0] for patch in h]\n",
    "plt.xticks(xticks_pos, feature_labels, fontsize=13,  ha='right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Year</td>\n",
       "      <td>2689.601562</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.365336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Origin</td>\n",
       "      <td>2137.199951</td>\n",
       "      <td>0.794616</td>\n",
       "      <td>0.290301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dest</td>\n",
       "      <td>1349.895386</td>\n",
       "      <td>0.501894</td>\n",
       "      <td>0.183360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>UniqueCarrier</td>\n",
       "      <td>508.218170</td>\n",
       "      <td>0.188957</td>\n",
       "      <td>0.069033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distance</td>\n",
       "      <td>320.266632</td>\n",
       "      <td>0.119076</td>\n",
       "      <td>0.043503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>FlightNum</td>\n",
       "      <td>259.619904</td>\n",
       "      <td>0.096527</td>\n",
       "      <td>0.035265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>DayOfWeek</td>\n",
       "      <td>93.597191</td>\n",
       "      <td>0.034800</td>\n",
       "      <td>0.012714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Month</td>\n",
       "      <td>3.603836</td>\n",
       "      <td>0.001340</td>\n",
       "      <td>0.000490</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        variable  relative_importance  scaled_importance  percentage\n",
       "0           Year          2689.601562           1.000000    0.365336\n",
       "1         Origin          2137.199951           0.794616    0.290301\n",
       "2           Dest          1349.895386           0.501894    0.183360\n",
       "3  UniqueCarrier           508.218170           0.188957    0.069033\n",
       "4       Distance           320.266632           0.119076    0.043503\n",
       "5      FlightNum           259.619904           0.096527    0.035265\n",
       "6      DayOfWeek            93.597191           0.034800    0.012714\n",
       "7          Month             3.603836           0.001340    0.000490"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# DRF Variable Importance\n",
    "data_rf2.varimp(use_pandas=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.axis.XTick at 0x10ec08860>,\n",
       "  <matplotlib.axis.XTick at 0x10de29e80>,\n",
       "  <matplotlib.axis.XTick at 0x10e9e3160>,\n",
       "  <matplotlib.axis.XTick at 0x10ede15c0>,\n",
       "  <matplotlib.axis.XTick at 0x10ede1fd0>,\n",
       "  <matplotlib.axis.XTick at 0x10ede3a20>,\n",
       "  <matplotlib.axis.XTick at 0x10ede9470>,\n",
       "  <matplotlib.axis.XTick at 0x10ede9e80>],\n",
       " <a list of 8 Text xticklabel objects>)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzkAAAFKCAYAAAA+O5FfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XvcZlVd///XG/EsIqKAjjaQiKLlKUPNzEkTRU38USKa\nqYnlTzEtswIyQTPB0lLzkJUiIIZoeUARQXHyiBhKmJCQOuOAzIhy8JTK4fP9Y62bubi47tPMPXNf\ns+f1fDz2476vtU9r77Wvfe3PXmuvnapCkiRJkoZih+XOgCRJkiQtJYMcSZIkSYNikCNJkiRpUAxy\nJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR9I2Jcn1Y8NPknwnyblJ/jnJ45JMPLclOW7C\n/D9K8tUkr01yp1nmWz1hvtHhHQvM+5p5lvPyzdk3i5XkM0mu2ZrrXEpJLun77a7LnZctKcnN+nae\nsdx5kaRtxY7LnQFJ2gQFHA0EuBlwB+C+wDOAQ4H/TPI7VXXxLPN+EDivf94deDzwEuCgJA+uqism\nzFPA8cCaCcs8b0LabPku4A3AVRPGr17gcpbKTH62Vdt6/iVJW4hBjqRtUlX91XhakjsD/wAcDJzZ\nA5bvTpj9A1V1wsh8twC+ANwP+EPgFbOs9p1V9anNzjy8vqq+tQTL0fYhy50BSdrW2FxN0mBU1eXA\n02g1IncHjlzgfD8DTqJdTD54S+VvsXozpRcmOTvJ93vTunOTPH+W6Z+T5N+SfD3Jj5NcleTTSZ42\nNt09klwP/Aqw41iTuTNG1j1rE6kk7xpvKjaz3CT/lGSfJO/tTQmvS/IrI9PdMclrklyY5P+SXJnk\njCSPXqJ9dn1f3u5J3plkfZIf9uZ5v9Knu22Sv0uytjd5/O8kB01Y3qF9eU9P8qQkn+vL+l6S9yT5\n+VnycZckb+1NFH+SZEOS9yV5wDzreHxvHnlVkp8lORT4Ga3G6jfGyurIkWUsqOxHpv9MX/6OSV6W\n5OKez7VJXp1k4k3QJPdOa/Y5ul3/keT3J0y7b5ITkqxL8tMkl/XjZu8J0+7ey+N/+v69sh8f70jy\nc5PyIklzsSZH0qBUVSV5FbCKFvC8ZJGL+OmSZ2oTJLk5cBrwaOBC4ERa3h4FvDmtlurQsdn+Efgy\nLchbD9yJ1hTvpCR7j9R+XUFr7ncosIJWczVTW/CNBWZxrqZi9wLOAb7a830b4Ad9u/ZkYxD6KeAj\nwE7AE4Ezkjynqo5fYB7mckfgc7RtfTewK+14OCPJw4G3A7elNV28ZR93SpKHVNW5Y8sq4KnAAcD7\ngLOABwJPAVYl+ZWq+vrMxD3w+SywG/BxWgD9c336JyR5clV9bMI6Dunr+AjwVto+Ohd4JfByWtmc\nMDLPaK3iQst+dH0AJwMPBU6nldETgMP7/nre6AxJntSnvznw0b5duwD3B14K/PPItE8A3ku7mXoq\n8PW+PQf1ffBrVfWVPu1tgM/3fXQmrUx2AFYCT+7rtOZT0uJUlYODg8M2MwDXA9fNM80taHe/rwNW\njqQf19OeOTb9rYHz+7jDJizvk33cccBR48Mi8v7Nvpy/n7CcF49N+6q+ra8DMpKeke04YGyevSas\n8+Y9/z8Bdhsb92ngZ7Pk9WZ9/WfMMv7Enoe7jqTdY6Z8ZtsvwGeAa4GDxtJ37mXwA2DXBe7PdRPy\ncLORPLxxbPpn93FXAf8G3Hxk3Ko+7j1j8xw6srzHjI374z7uo2Ppn+jTv3Qs/eF92zcAt5qwjmuA\nRy22LDaj7K8HzgZuP5J+G1pA8rPRcqAFbD8A/g942IR1jZbBHfs+Xg/cc2y6XwB+CHxhJO3JPS/H\nzrINt13od8zBwcFhZlj2DDg4ODgsZmABQU6f7rJ+ofngkbSZ4OD9bAwu3gys7ekfGr3wHZnvkyMX\nuuPDtYvI+zdnWcZ1wDdGptuBVgPxLUYCnJHxu/b8vGuB631KX8chY+lbKshZB+w4YZ4H9fEnzbLM\ng/oyn7vA7ZoryLkauPXY9DvSgozrgBUTlvct4GtjaTMByGmz7KNv9OXdpaet7NP/L7DDhHnePV4W\nI+s4eVPKYjPK/jrgERPmeVUft/9I2p/3PPzNAtb5J33+359l/Bv7+L3755kg5+jFbp+Dg4PDbIPN\n1SQN1Uzzq/EmVQGe1IdRHwEOrKrZmmAVsKqqPr0EeduzqtbNMX5fWo9x64GXJzd57jy0u/P73igx\nWUm7GH0UrWnQrUdGF61p2tZwXlVdOyH9Yf3vLkmOmjB+D9q27Tth3GL9T1X932hCVV2b5HJa8HHp\nhHkupXU+MclNOpyoquuSfJYW2DyQFlg/cGb6qrp+wnLOojVLeyCtGdaoL862MfPZjLIfb5oHLXiE\n1hRtxkP6ck5fQHYe2v8+aJZynnkmZ19aMPhJ2r57WZL9aM00P0s7juw9T9ImMciRNDhJbklrMgNw\n+djoAp5dVSemRQ8/D/wV7cLzbcAfzLXopcriPON37X/vRXsWYza3vWGByT1oF8k70S7IT6fVZlxH\n28bfpT17sjWsnyV9Zrse24dJipHt2gxXz5J+La3J1Wzjbj5LnjbMMs/Mtu489veyWaafSb/DhHXM\ntt/mtBllf11V/XhC+kyAerORtJn8TgoOx+1KO8bn+i4B3A6gqq5O8hDac2K/STs2Alye5M3AX1fV\ndQtYryTdwCBH0hA9gnZ+W1+Tu2oOtE4KgK8n+R1gL+DQJB+sqo9svaxONHOB/t6qOmSB8/wp7QL7\nGVX1r6MjkjwDeOYi8zBTCzHb78T4Rfqo2e6+z2zXYVX1j4vMz3IK7X1Kk+zR/1499nePCdMC3GVs\nulGbWmux1GU/ycx7nVYAX5tn2qtp23KfqppvWgCq6hLguQBJ7kOrkXohLfAp2o0ISVowu5CWNCi9\nduYvaBdGJy1knh7svJh2Mfs3mdA+bCv7Ku0h74clWeh5+h79779PGLeKyRfQ1zFLrVLfJ9+nNX26\nkSQ3o/WotVhn97+P2IR5l9sjxxP6fnh4//jlsb+PmOU4ehStLL60iHXPBJw3m2X8ppT9Yp1NO1YO\nWMS0v7YpK6qqC6rqTcDjetKTN2U5krZvBjmSBiPJbsB7aBeka4FjFjpvVZ0DfBi4N0tz53uT9edZ\n3kQLMN7Qm9/dSH8Py71Hktb0v6vGpns8rVexSb4H7JBktuc1zgF+PsmqsfSjgbvNugGzqKov0LoK\nPjjJxH2c5H5Jdp00bpntn2S8id0fAXvSOgS4DKCq1tKeMbkH8KLRiXvX1QcD36V1k7wgPeC8ktbF\n8iRr+t9VY+ubq+wX6zhar2gvzMg7j0bWNXoMvZ0WIL8yyS9NmHaHJI8c+XzftBf5jpupDfvRZuVc\n0nbJ5mqStkkjDzTvQGs6dV/gV2nPVJxNa7pzxSIX+3Lae0KOSnLSLA/Pby1HAb8IvAA4MMlZwLdp\nzabuSXuR558B/9OnfzMtOPtAkvfSnu/4BWB/4BTaM0fjPgH8f8AHk5xOe1blm1X17j7+tbT39JyW\n5GTahfbD2fiOm02pkTmE9u6Y45L8ES2QuooWND2A9jD6L9MCsGlyKnBqkn+j9aj2INqzI5fTmlWN\neh6t97K/S3IA7eH+lcBv07qJfvZ4pwjM/5zWJ4DfSvIBWm3RtcDqqvosm1b287lRfqrq8t6s8z3A\np5KcBnyF1kzu/rQupu/Vp/1ukqfQ3il0TpKPAxfQapR+jnbs3g64fV/844BjknwOuIi2T+8OHEir\nbfzbTci/pO2cQY6kbc1M05uZB/J/RmvatRY4HnhfVZ25SQuuOi/J+2kX/s+jXTyOr3dzLWg5PcB6\nUpLfBZ5Fe1nm7WgXgN8AjmSkd66e91+nPbvwBFrTpv+iXSj+hPYyy/F1v40WXDyV9lzHjrSL6Xf3\nZZ6R5CDgZbSXZf6Q9lD7bwHHzrItNdc2VtW6fnf/RbQuo59OC1TX0y6E/67/XahF52ETxhUtWPgX\n2n5/Iu24OwU4sqpu9ALVqvrfvo0vo72QcxWtZuNU4JiqmtRUbb7j4oW0AOnRtPLdAfhL4LObWPbz\nrfMm46rq1CQPpr0s9FG0IOpK2stqjx2b9swkMy8J3Z8WEP+U1vHCx2gB0IzTgLv2aQ6kdaCwvqf/\nfa9llaRFyXy9M/ZmEp+ivVzvFsAHq+rIJLvQ7uispFWVH1xVV/d5jgCeQ7vT9OKqOqOnPwh4J3Ar\n2jsH/mgLbJMkSUsiyaHAPwG/O1LDJUmacvM+k1NVPwV+vaoeSHt/wKN6u+LDgY9X1b1o/f4fATf0\ninIwrcnBAcBbRh6+fCtwaFXtA+wzoX2zJEmSJG2WBXU8MNKP/i37PFfSqpSP7+nHs7H3kyfR3tp8\nbVWtAS4G9kuyB7BTVc287OwE7DFFkjT9lru3PUnSIi0oyOk9oXyZ1kZ2dVVdAOxeVRsAqmo97aFD\naH3oj77J+9KetgK4ZCT9Erbe27clSdpUS/U8liRpK1lQxwNVdT3wwCS3Bz7WuxMdP+n7IyBJGpSq\nejutS2RJ0jZkUb2rVdX3e7eRDwY2JNm9qjb0pmjf6ZNdyo1fHne3njZb+k0kMWCSJEmSNKeqmtik\neN7maknulGTn/v+tgcfQ+uj/EBtfMvYsNr7Y7EPAIUlukWQvYG/gnN6k7eok+/WOCJ7JHC9Dq6qp\nG4466qhlz8P2PlgG0zFYDtMxWA7LP1gG0zFYDtMxWA7LP2xvZTCXhdTk3AU4vgcmOwAnVtUn+jM6\npyR5Du39FAf34OSCJKfQ3nNwDfCC2piLw7hxF9KnL2D9kiRJkrRg8wY5VfUV2pudx9OvAH5jlnmO\nAY6ZkH4u7Q3ekiRJkrRFLKh3NTWrVq1a7ixs9yyD6WA5TAfLYflZBtPBcpgOlsPysww2ynzt2ZZD\nkprGfEmSJEmaDkmoTe14QJIkSZK2JQY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mS\nJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNi\nkCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmS\nJEmDYpAjSZIkaVAMciRJkiQNyo7LnYHZJFnuLGxxu69cyfo1a5Y7G5IkSdKgpKqWOw83kaSYwnwt\nuYRp3P+SJEnStEu7lp5YM2JzNUmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5\nkiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDcq8QU6SuyU5K8lXk3wl\nyR/29KOSXJLkS3143Mg8RyS5OMmFSfYfSX9QkvOTXJTk9VtmkyRJkiRtz1JVc0+Q7AHsUVXnJbkd\ncC5wIPBU4AdV9Xdj0+8LvBv4ZeBuwMeBe1ZVJfkC8MKq+mKS04A3VNXHJqyzmCdfg5Aw3/6XJEmS\ndFNp19KZNG7empyqWl9V5/X/fwhcCKyYWfaEWQ4ETq6qa6tqDXAxsF8Plnaqqi/26U4AnryoLZEk\nSZKkeSzqmZwkewIPAL7Qk16Y5Lwk/5Jk5562Alg3MtulPW0FcMlI+iVsDJYkSZIkaUnsuNAJe1O1\n9wEvrqofJnkL8MreDO1VwOuA5y5Zzo4+euP/q1a1QZIkSdJ2afXq1axevXpB0877TA5Akh2BDwMf\nrao3TBi/Eji1qu6X5HCgquo1fdzpwFHAWuCTVbVvTz8EeGRVPX/C8nwmR5IkSdKsNuuZnO4dwAWj\nAU5/xmbGQcB/9/8/BByS5BZJ9gL2Bs6pqvXA1Un2SxLgmcAHF7ktkiRJkjSneZurJXk48DvAV5J8\nGSjgSODpSR4AXA+sAZ4HUFUXJDkFuAC4BnhBbayuOAx4J3Ar4LSqOn1Jt0aSJEnSdm9BzdW2Npur\nSZIkSZrLUjRXkyRJkqRtgkGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJ\nkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEka\nFIMcSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORI\nkiRJGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnS\noOy43BnQdNpjzz3ZsHbtcmdjq9h95UrWr1mz3NmQJEnSEklVLXcebiJJMYX5WnIJ07j/AZKwXZQB\nTHU5SJIkabK0a7hMGmdzNUmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJ\nkgZl3iAnyd2SnJXkq0m+kuRFPX2XJGck+VqSjyXZeWSeI5JcnOTCJPuPpD8oyflJLkry+i2zSZIk\nSZK2ZwupybkWeElV3Rd4GHBYknsDhwMfr6p7AWcBRwAkuQ9wMLAvcADwliQz/Ve/FTi0qvYB9kny\n2CXdGkmSJEnbvXmDnKpaX1Xn9f9/CFwI3A04EDi+T3Y88OT+/5OAk6vq2qpaA1wM7JdkD2Cnqvpi\nn+6EkXkkSZIkaUks6pmcJHsCDwDOBnavqg3QAiFgtz7ZCmDdyGyX9rQVwCUj6Zf0NEmSJElaMgsO\ncpLcDngf8OJeo1Njk4x/liRJkqStbseFTJRkR1qAc2JVfbAnb0iye1Vt6E3RvtPTLwXuPjL73Xra\nbOmTHX30xv9XrWqDJEmSpO3S6tWrWb169YKmTdX8FTBJTgC+W1UvGUl7DXBFVb0myZ8Du1TV4b3j\ngZOAh9Cao50J3LOqKsnZwIuALwIfAd5YVadPWF+xgHxt8xIWsv+XQxK2izKAqS4HSZIkTZZ2DZeJ\n4+a7uEvycOBTwFdoTdIKOBI4BziFVjuzFji4qq7q8xwBHApcQ2vedkZP/yXgncCtgNOq6sWzrNMg\nZ5kZ5EiSJGmabVaQsxwMcpafQY4kSZKm2VxBzqJ6V5MkSZKkaWeQI0mSJGlQDHIkSZIkDYpBjiRJ\nkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUAxyJEmSJA2K\nQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAjSZIkaVAMciRJ\nkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQ\nDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJ\nkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBmXeICfJ25NsSHL+SNpRSS5J8qU+PG5k\n3BFJLk5yYZL9R9IflOT8JBclef3Sb4okSZIkLawm5zjgsRPS/66qHtSH0wGS7AscDOwLHAC8JUn6\n9G8FDq2qfYB9kkxapiRJkiRtlnmDnKr6DHDlhFGZkHYgcHJVXVtVa4CLgf2S7AHsVFVf7NOdADx5\n07IsSZIkSbPbnGdyXpjkvCT/kmTnnrYCWDcyzaU9bQVwyUj6JT1NkiRJkpbUjps431uAV1ZVJXkV\n8DrguUuXLeDoozf+v2pVGyRJkiRtl1avXs3q1asXNG2qav6JkpXAqVV1v7nGJTkcqKp6TR93OnAU\nsBb4ZFXt29MPAR5ZVc+fZX3FAvK1zUtYyP5fDknYLsoAprocJEmSNFnaNdykR2gW3FwtjDyD05+x\nmXEQ8N/9/w8BhyS5RZK9gL2Bc6pqPXB1kv16RwTPBD64yO2QJEmSpHnN21wtybuBVcCuSb5Fq5n5\n9SQPAK4H1gDPA6iqC5KcAlwAXAO8oDbeIj8MeCdwK+C0mR7ZJEmSJGkpLai52tZmc7XlZ3M1SZIk\nTbOlaK4mSZIkSdsEgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIG\nxSBHkiRJ0qAY5EiSJEkalB2XOwOSZrfHnnuyYe3a5c7GVrH7ypWsX7NmubMhSZIGIFW13Hm4iSTF\nFOZrySVM4/4HSMJ2UQZgOUyLKS4HSZI0fdKuHTJpnM3VJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmS\nJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0\nKAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciR\nJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKk\nQZk3yEny9iQbkpw/krZLkjOSfC3Jx5LsPDLuiCQXJ7kwyf4j6Q9Kcn6Si5K8fuk3RZIkSZIWVpNz\nHPDYsbTDgY9X1b2As4AjAJLcBzgY2Bc4AHhLkvR53gocWlX7APskGV+mJEmSJG22eYOcqvoMcOVY\n8oHA8f3/44En9/+fBJxcVddW1RrgYmC/JHsAO1XVF/t0J4zMI0mSJElLZlOfydmtqjYAVNV6YLee\nvgJYNzLdpT1tBXDJSPolPU2SJEmSltRSdTxQS7QcSZIkSdosO27ifBuS7F5VG3pTtO/09EuBu49M\nd7eeNlv67I4+euP/q1a1QZIkSdJ2afXq1axevXpB06Zq/kqYJHsCp1bVL/bPrwGuqKrXJPlzYJeq\nOrx3PHAS8BBac7QzgXtWVSU5G3gR8EXgI8Abq+r0WdZXLCBf27yEhez/5ZCE7aIMwHKYFlNcDpIk\nafqkXTtk0rh5a3KSvBtYBeya5FvAUcCxwHuTPAdYS+tRjaq6IMkpwAXANcALauNVy2HAO4FbAafN\nFuBIkiRJ0uZYUE3O1mZNzvKzBmE6WA6SJEmTzVWTs1QdD0iSJEnSVDDIkSRJkjQoBjmSJEmSBsUg\nR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJ\nkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgG\nOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJ\nkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEEx\nyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg7JZQU6SNUn+K8mXk5zT03ZJckaS\nryX5WJKdR6Y/IsnFSS5Msv/mZl6SJEmSxm1uTc71wKqqemBV7dfTDgc+XlX3As4CjgBIch/gYGBf\n4ADgLUmymeuXJEmSpBvZ3CAnE5ZxIHB8//944Mn9/ycBJ1fVtVW1BrgY2A9JkiRJWkKbG+QUcGaS\nLyZ5bk/bvao2AFTVemC3nr4CWDcy76U9TZIkSZKWzI6bOf/Dq+qyJHcGzkjyNVrgM2r888IcffTG\n/1etaoMkLYM99tyTDWvXLnc2trjdV65k/Zo1y50NSZImWr16NatXr17QtKnatBjkJgtKjgJ+CDyX\n9pzOhiR7AJ+sqn2THA5UVb2mT386cFRVfWHCsoolytdUS1iq/b/UkrBdlAFYDtPCclh+U1wGkiSN\nS/vdmviM/yY3V0tymyS36//fFtgf+ArwIeDZfbJnAR/s/38IOCTJLZLsBewNnLOp65ckSZKkSTan\nudruwPuTVF/OSVV1RpL/BE5J8hxgLa1HNarqgiSnABcA1wAvKG8ZSpIkSVpiS9ZcbSnZXG35bTfN\nc8BymBaWw/Kb4jKQJGncFmmuJkmSJEnTyCBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUAxy\nJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDsuNyZ0CS\npPnsseeebFi7drmzsVXsvnIl69esWe5sSNI2LVW13Hm4iSTFFOZrySVM4/4HSMJ2UQZgOUwLy2H5\nWQbTYYrLQZKmSdr5MpPG2VxNkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUOxdTZIkLYi93Ena\nVti72nKa4h507MloOlgO02G7KQfLYDpYDtNhistBUmPvapIkSZK2GwY5kiRJkgbFIEeSJEnSoBjk\nSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ\n0qAY5EiSJEkaFIMcSZIkSYOy43JnQJIkSQu3x557smHt2uXOxha3+8qVrF+zZrmzoW1Uqmq583AT\nSYopzNeSS5jG/Q+QhO2iDMBymBaWw/KzDKaD5TAdLIflN8VloOmQdoxk0jibq0mSJEkaFIMcSZIk\nSYOy1YOcJI9L8j9JLkry51t7/ZIkSZKGbasGOUl2AN4EPBa4L/C0JPfemnnYLKtXL3cOZBlMB8th\nOlgOy88ymA6Ww3SwHJbdasvgBlu7Jmc/4OKqWltV1wAnAwdu5TxsOg+c5WcZTAfLYTpYDsvPMpgO\nlsN0sByWnUHORls7yFkBrBv5fElPkyRJkqQl4XtyJEmSpEWY5ncVveIVr1jS5W2r7yvaqu/JSfJQ\n4Oiqelz/fDhQVfWasensFF2SJEnSnGZ7T87WDnJuBnwNeDRwGXAO8LSqunCrZUKSJEnSoG3V5mpV\ndV2SFwJn0J4HersBjiRJkqSltFVrciRJkiRpS9vqLwPV9iPJEUn+aamn1fYtyVuT/MVy52OpJDkt\nye8udz6mydDKeBokWZnk+v6+ukUdd0k+meQ5WzaHWowkt0pyapKrkrxnK6/b42GZ9e/yzy93Pqbd\nNh/kJDnAhKmoAAAPpklEQVQxyTvG0h6Z5LtJdl+ufA1RkmcnOT/Jj5J8O8lbkuw82/RVdUxV/cFC\nlr2YaTVZkjVJfpzk6iRXJPlMkuclmfhA3iKW+80kj1rCfN7k5JzkqCQnLmT+qnp+Vf31UuVnLkn2\nS/KRJFf2c8rZSZ69lOuoqsdX1YK2fSjmO1YXWsZLfWwOxcj+/X6SHyT5PnBX4IamG0t13I0HTz3t\nWT3tpWPTrkvya5u7zmmwpc63I8v/0yQX9d/bNUleneQWI5P8NnBnYBfg6b2cf3lk/t/pZTCe5iMC\nS6yXz0+S3HEs/cu9DH5uM5c/Kai0GdYCbPNBDvBi4HFJHg2Q5JbAPwF/XFUblmoloyfw7VGSPwGO\nAf4EuD3wUGAlcGaSmzzb1TuZ0NZVwBOqamda2RwL/Dnw9mXN1U3NdnKeqpN2kocBnwA+Cdyjqu4E\nPB947CYu7yYXP5tzQbSNf8e2lWN1WzWzf29fVTtV1e2Bb2+hdaWvb/xYvgL4syS33ULrXW5b7BhO\n8g/Ac4FnADsBB9A6bDplZLKVwEXVXAd8DhgNIB8BXDgh7T82N3+6iQK+CTxtJiHJLwC3Zsv9ri1J\nMD102/yFe1VdAbwI+KcktwGOBv63qk5Mc2SS/03ynSTvnql56OPem+SyfhfmrCT3nlluryF6U5KP\nJvkB8KvLsX3TIMlOtP36wqo6s6quq6pvAQfTTrTP6Hfi39v321XAs8bvzid5Zr/jcXmSl43ehR2d\nduTO4DOTrO1ld+RW3/Bt08yd8B9U1YeBp9LK4j5JbpHktX2fXpZWE3dLgCS7pjV9uDLJ95L8R08/\nAfg54NR+V/ils614sXmcdWSriV2X5CVJNiS5NCO1J0mOS/LKkc9/mlazeEmS38tITdH4HbB+h/nT\nI5/vneSMvs0XJnnKSFb+Bjiuql7bzzNU1Zer6ml93jv0ffadPv+pSVaMLPuTSV6Vdof3R8Bec6SN\n5vE5SS7oy/zo6F3Avm0vSHIRcNHCd/lUmnSsPrMfqzeU8WKPzSSn9OP7yiSrk9znhhW25b4pyYf7\nPJ9PstfI+PuOHA+Xpb3mYOb34vC035LLk5yc5A5ba0dtovm+Zzccd0l2SPK6vm1fT3JYxmpngD37\ncfv9JKdn413rmYvmq/q4h/TPFwKfp90Ym7T+8e/xI5OsG/n8zSQvTWs98P0k/5Jkt7Rmdlf3cpq1\nJcFWMtf59vFJvtTzujbJUTfM1I6/w260oOS/khyYZG/azZSnV9U5VXV976Dpt4DHJlmV5Gjg5cAh\nfd/8HvBpbhrQvAZ45Fjap/r65jymkzw0yWf79+jLSUaXM5rvu/S8Tyzn7ciJwLNGPj8LOH7mQ5Lb\nJzmh/158MyPNcdN/l5L8bdr16NeTPLaPexWt3N7Uy/qNI+t4TFpt3xVJ3rRlN2/btM0HOQBV9T7g\nS8C/0u5+/H4f9RLaHZBfBe4G/BD4h5FZTwXuAewB/DftIB31NOCoqtqJdrLeXv0KcEvg/aOJVfUj\n4KPAY3rSk4BTquoOwLtnJgPoFxpvpu3TuwA705pP3GiRY58fDtwT+A3g5UnutRQbsz2pqi8Cl9BO\nkscCewP3639X0H4ooV2IrAN2BXYDjuzzPxP4FvDEflf4tVsp63vQ7mDelfadfvOkC5okj6N9zx/N\nxmNlvjtnM8fkbWg9Pb4LuBNwCPCWtMDn1sDDgH+bYzk7AO8A7k672P4xMP5D84ye/51o+3G2tJnt\nORA4HHgyrSnKp2nntVEHAr8M3IcB6cfqpbRjddRij83TaOf13Wi/CyeNLe+pwFHAHYCvA38NkOR2\nwJl9/rvQviOf6PO8iHZ+ewTtmLwSeMvmbvMU+QNaDeX9gAfRjr/x79HTaBdud6b9Hszc8Ji5sL59\nL4cv9M8F/CXwR4sICMfXeRDwKOBewG/Sfm8O73m4Ga1cpsbY+faHwO/2mp4nAP9/kif1SY8Hbnge\nKsn9acfVR2jnsnVVde7Ysi8BvgA8pqqOBl4NnNz3+XG04OXhfXl3Am5Dq/nZbyRt3z4dzHFMp92s\n+TDwyqrahVbW/5Zk19E8JdkTWA28sapetwm7bEjOBnZKcq9+c+CptN+WGW+infP3BFbRbuj83sj4\n/Wg3BnYF/pb220JVvYz2O/DCXtajx/wTgF8C7g8cnGT/LbBd27RBBDndYbST4SuqaqZa/nnAkVW1\nvqp+BvwV8BRobyCtqhOq6sd93CuBX+oXNzPeX1Xn9Omv2WpbMn3uBHy3qq6fMO6yPh7g81V1KkBV\n/WRsut8CPlRVn6+qa9l4cT2bor049mdVdT7wX7Qvshbv27QT5x/QmnFe3QPUY9lYvX4N7cJur15T\n99mxZWztqvGfAX/V8/JR2gXDpCD3KbTalgur6v9oNY4L9UTgm/08UFX1X7Sg5im0du470I7viarq\niqp6f1X9tO/PY7jxnVSAd1bV//S7sdfOkTbjecAxVXVR/74dCzwgyd1Hpnl1L8OfLmJbtxXfBu44\nlraoY7Oq3tnP69fQzuv3T6uNnvH+qjq379+TgAf09N8ELquq1/fzzo/6RSu0cvmLqrpsZLm/nelu\nxvyBfof3iiT/Ps+0TwHe0LfvatpxN+64qvp6P+5OYeN+m3GTc0Q/d59Ja8a1Kf6hqr5bVZfRLvTO\nrqrz+2/2+4EHbuJyt6RvA3esqk9V1VcBquq/gZPZWKvyIeCeSe7RPz8DeE8/H9yJ2c87o7+3474A\n3CbJL9Ju7H6m/w5/YyTtmz1YgrmP6d8BPlJVH+v5/wTwn8DjR9Z3X1pT3r+sKpuZNjO1OY+hBSwz\n16I70oKew/u5aS3wOkYCXWBtVb2jqooWBN8lyW7zrO+YXou4jlYW49/J7d40n6AXpaq+A3wXuGAk\neaYpwxVJrgDOB67vVd47JPmbXi14FXAx7cJ69ASyDkHbr3ea5Qf9Ln08zL2/7jo6vl+Qfm+e9Y4+\nU/Vj4HbzZ1UTrKDd9bwNcO7I9+GjtOAH2p2jrwNn9OYLm3pRshDXATcfS7s57WJ2xvfGgurZyv9G\nxxWwloUHZCuBh45cCF4JPB3YnXZX83ra8T1RklsneVtaE8yraM127pDc6DmbSd+Jub4nK4E3jJTR\n92jnpRUj01wycc5hWEF7lmPUgo/Nfl4/tk93Fa2d/Ph5ff3I/6PH1d36eiZZCbx/pFwuoB2v09y5\nzYFVdcc+HDTPtOPfo0nH6Gz7bT4vB56/gAu2SUZ/A/5vwudp/E1YAVyR1mnJWb150lW0oOJOACOB\n4jP6+eJpwAl9/u8y+3ln9Pf2Rvoyz6EFUr9GCwoBPjuS9qmRWeY6plfSagZGz40Pp9Wwz3g67Vw0\nV2339uZdtP3ybDaWJ7Ry35Eb19yv5cbn9Ru+X/36COY/vr1GmsdggpxZrKNV7c6c6Hepqtv2gOiZ\nwOOAVb151d60i6PRC5SpehB6GX0e+Cmt6cANevOOA9jYpGOu/XUZ7SJiZt5bs/ECW1tIWs86dwU+\nQDsJ3nfk+3CH3pSCqvphVb20qu5Ba8LwkiS/3hez1N+Db9Gq7EftRTvpL9ZltOZiM1Zy4/z+iBbc\nzRj9kV4HrB47P9y+ql7Yf2Q+T6uBnM2f0JrI/XI/h8zU4sx3Dplrf64DnjeWp9tV1dkLnH+bNXKs\nfmY0fZHH5tNpNTKP6mWyJzc9r89mHa2Z2yTfAg6Y8Fsya03fFFhM7euNzs+0G4QLNefxWFVfA/4d\n+Avm/m7OekNhWzF2DL+bdt5d0Y/Ft3HjMjmBVoPzaOBHM61GgLOAuyd58Niy707r8Ofjc2Rh5rmc\nX2VjkDOTdsPzON1cx/Q64ISxcTtV1d+OzH80LeD617EbO9utas8qf5N2XTRae3o5LYBcOZK2ktY8\nd0GLXpIMboeGHuS8DThmpqlHr8H5zT5uJ9qF+5Vpvb+8Gg+kiarq+7Sq7H9I8tgkO/a2uO+hnSgX\n0g3p+4Df7A8z3pz5mxV50twMSXZK8kTa8xwnVtVXgH8BXp/kzn2aFTNteJM8YaTpxA+Aa2k1LtDu\nFi1lf/zvAV7W158kv0FrOva+TVjWKcCzk+zbn7EZbwZ5HnBQr3XZGzh0ZNyHgX2SPKMf0zdP8uBs\n7IDkz/qy/yT9Iesk908y84zMTrS7yd/v44/ehPyP+0fgyP4MG0l2TvLbS7DcqTXhWP3q2PjFHJvj\n5/VjWPh5/cPAHklelNZJx+2S7NfHvQ14dXonEEnuPPJ8xbZktvPqKcCLk9w17fmZP1vEMi+n1XrO\nFiBC+/34PdpzUDPOAx6fZJcke9B6St0mzXIM3w64sqqu6cfR00fnqarP047N1zHyG1pVF9OOt5OS\nPKTXTt6Xdn48o6o+OUdWPgX8OnD3ap0VQKvJWUVr7j0a5Mx1TL+L9nu9f1//rdI6hhh9jvYaWjPH\n2wInGujc4Dm0mywztTGhna/eC/x1P6+sBP6YhV07wdL/Bm83hhbkjP+YvY7WJOcTSa6m3V2ZuTty\nHO3u1beBrzB293DCsrZr/Q7OkcBrgatpd7nXAr9RC3heqaouAP6QdoH7beD7wHdoFyQTZ5nnsyY7\ntR/r3wKOoJXXTM9dfwb8L3B2bz5xBrBPH3dP4ONpPQl+FnhzVc38IB4D/GVvtvCSJcjjK2ndnX6G\n1jTpWFpPQhfMMc/E8q+q04HX0+5+XsTGWsUZf0/7MV5P+86/a2TeHwL70zoc+HYfjgVu0cd/nvac\n36OBryf5Li0I+UhfxOtpd6K/27fntAXkec60qvpAz8PJvYzOp9U4zzX/tmquY3XUYo7N4/vyLqV1\nJvO5hWamHw+PodUWracdT6v66DcAH6Q1mbu6L3e/CYuZFgvppn30/3+mnQ/OB86lHePXjjQZnfW4\n6xdzfw18tpfDTfZLVa2hXdCNdid9Yl/fGuB02jMrc23DNB77cx3DLwD+qo9/Ge23b9wJwC9w4wfU\nqarDaDel3kUL7E+jnePmu+HxOdorHm6o+a2q79EC0Q1VNdocc9Zjuj+3cyDtN/9y2m/9S9l4zVh9\numtpLTx2Y/vu/n30HP7NqvrShHF/SGtN8Q1asPmuah1GzLtMWlk9Ja3Xx9dPGD/ps4C0Z5ykravf\nZb0K2Ls/hCctiSTX046rbyx3XqRtUVqvhW+tqr3mnVibLMkzgD+oqkG8IFWaNkOrydEUS/LE3mzo\ntrRatvMNcCRpefXmSAckuVla98FHceNnCrTEevPaw2jNxiRtAQY52poOpDUJuoTWfvuQ5c2OBsrq\naWlxAryC1nz0XOCrtEBHW0B/FvI7tCbz4+/BkrREbK4mSZIkaVCsyZEkSZI0KAY5kiRJkgbFIEeS\nJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQ/h/RhNi18z1tWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e952588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot DRF Feature Importances\n",
    "importances = data_rf2.varimp(use_pandas=True)\n",
    "feature_labels = list(importances['variable'])\n",
    "Index = importances.index\n",
    "\n",
    "plt.figure(figsize=(14,5))\n",
    "h = plt.bar(range(len(feature_labels)), importances['relative_importance'],width=0.6, label=feature_labels, color ='aqua')\n",
    "plt.title(\"DRF Feature Importances\", fontsize=20 )\n",
    "xticks_pos = [0.65*patch.get_width() + patch.get_xy()[0] for patch in h]\n",
    "plt.xticks(xticks_pos, feature_labels, fontsize=12,  ha='right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Origin</td>\n",
       "      <td>4716.587402</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.369159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Year</td>\n",
       "      <td>3886.308838</td>\n",
       "      <td>0.823966</td>\n",
       "      <td>0.304174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dest</td>\n",
       "      <td>2975.810547</td>\n",
       "      <td>0.630924</td>\n",
       "      <td>0.232911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>UniqueCarrier</td>\n",
       "      <td>469.562317</td>\n",
       "      <td>0.099556</td>\n",
       "      <td>0.036752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DayOfWeek</td>\n",
       "      <td>339.474030</td>\n",
       "      <td>0.071975</td>\n",
       "      <td>0.026570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>FlightNum</td>\n",
       "      <td>280.978149</td>\n",
       "      <td>0.059572</td>\n",
       "      <td>0.021992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Distance</td>\n",
       "      <td>107.865120</td>\n",
       "      <td>0.022869</td>\n",
       "      <td>0.008442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Month</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        variable  relative_importance  scaled_importance  percentage\n",
       "0         Origin          4716.587402           1.000000    0.369159\n",
       "1           Year          3886.308838           0.823966    0.304174\n",
       "2           Dest          2975.810547           0.630924    0.232911\n",
       "3  UniqueCarrier           469.562317           0.099556    0.036752\n",
       "4      DayOfWeek           339.474030           0.071975    0.026570\n",
       "5      FlightNum           280.978149           0.059572    0.021992\n",
       "6       Distance           107.865120           0.022869    0.008442\n",
       "7          Month             0.000000           0.000000    0.000000"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# GBM Variable Importance\n",
    "data_gbm2.varimp(use_pandas=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.axis.XTick at 0x10e799080>,\n",
       "  <matplotlib.axis.XTick at 0x10ee14f28>,\n",
       "  <matplotlib.axis.XTick at 0x10ddebb00>,\n",
       "  <matplotlib.axis.XTick at 0x10ee5bc50>,\n",
       "  <matplotlib.axis.XTick at 0x10ee606a0>,\n",
       "  <matplotlib.axis.XTick at 0x10ee640f0>,\n",
       "  <matplotlib.axis.XTick at 0x10ee64b00>,\n",
       "  <matplotlib.axis.XTick at 0x10ee68550>],\n",
       " <a list of 8 Text xticklabel objects>)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzkAAAFKCAYAAAA+O5FfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8bVVd9/HPF/EuKoncjggEgmhZWR41U49XRA18SknM\nRKUyxUuPmoE3SMtL6hOaQVkqNwvR8o6It5OpoKYSFiSEcOR6FBDwloL8nj/G2Jx1Fmvvs/c5+5y9\nztyf9+u1XnuvMcecc6w55pxr/uYYc6xUFZIkSZI0FNssdQEkSZIkaTEZ5EiSJEkaFIMcSZIkSYNi\nkCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSthpJ7pXk/yX5apKrk/y0/z0ryZuS3H/C\nPEcluWns9bMk1yb5QpLnJbnVhPlWj+R/5hxlGl3+uxbwWcbLNP56xrw3zCJIcmmS87fkOhdLklv1\nbfbTpS7L5pZkr/5Z37HUZZGkabbtUhdAkuYjyVHAq4AAXwNOAa4BtgPuBzwfeHGS51fVcRMWsbq/\noJ37dgMOBN4OPBj4vbH81V83Ar8PHD+hTAGeBdzAxp1PCzi6f6ZxZ2/E8jaFvwwtSRoMgxxJU68H\nOEcBa4BDquqsCXl2AP4YuMssi1ldVa8Zm+cewHnA05K8oqq+PWG+jwJPSrJfVZ03Nu1xwD2BfwF+\nayGfaUZVvXZj5tOyNSkgliSNsbuapKmWZE/gFcBPgAMmBTgAVXVVVb0S+Mv5LruqLgW+2d/efZZs\n/0C7sPyDCdP+APgh8J75rnNjJblDkpcnOTvJD5J8v3e3O3hC3tskeUGS05JcnOR/e7e+M5I8dizv\no5LcBOwK7D3WZe4dPc+cXaSSfD7JDZOW28v8wF6Wq3tXwV1H8t0jybFJLuzlvCrJByd1PdyIbXZz\nuZPsneSf+/KvT3J6kvv0fDsmeWeSy5P8OMmXkjxswvL+vC/v15M8K8nXk/woyZVJ/j7JjrOUY58k\nJyW5LMlP+t/jk/z8Btbxe70s309yQZLXAufTWt1+f6yuntbnn3fdj6zz0iTn933sLUnW9PnOT/KS\nObbvg5KcOva5Tk/y2xPyPrhv/yt73m8nOS7JzhPy/nySf0jyP337XpXknL6fzHYTQ5LWY0uOpGn3\nbNq56pSq+u8NZa6qm+a74CS7AfsC17Mu2Bn3TeBzwNOT/GlV3dDn3Ql4InBSn3+zSXJXWle7XwS+\nCryTdpPqccApSe491kp1d+CvgC8AZwDfBXahdc/7eJJnVdWJPe+3aF3mXkLrdvdW1rUWfG2eRZzp\n2jfJQ2mtcP9KCxh37Oshya8Bn6C1vp0OvL+X/f8AX0zyxKr61DzLMJe9gC8B3wDeDexJa3n7bA9m\nTgeupnWBvBtwCG073auqLp/wOV8GPBp4L3Aa8DDgMGBVkpVV9b2ZGZI8iFYHdwA+BPw3cG9a98gD\nkzyyqs6esI4jgEcBHwE+Ddyp/90OeCGtbj48Mt85/e9C6n50nbcBPtXn/xjwM1o9vCnJbarq9aMz\nJPkjWlfPn/Zy/A+tbh8APAf455G8fwAcB/yo570U2Id2k+CJfZtd0fPuCvx7316nAe8Dbk+rs98D\njgGuQ5I2pKp8+fLla2pftAu7nwHP2sj5jwJuAj7b/z8KeC3tYvcq2kXgQRPm+2xf788Dv9uX8Tsj\n04/o0x9Muxi9CXjXAsp1U5//qAmvQ8fyntzzvmgs/ba0C9kbgfuOpe8yYZ13Bs4F1gK3Hpt2CXD+\nLGXdq5f3HbNM/zfgp2NpM9vkZ8AzJ8yzLS3A+iHw4LFpuwKXA98Gtp3HtrxVX9d4GfYaKcNLx6Yd\n3addC7xtbNoz+7Q3jqW/tqf/CPiFsWlv69OOG0kLreXlZ8CTx/If0vOfM8s6rh9fxzzrYmPr/mfA\nB4HbjqTvRAsorgIykv6LtED1O8A+E9a1YuT/e9NaYc8FdhzL9+i+7753JO2Pe1n+aMJy7wDcZr7H\nmC9fvpb3y+5qkqbdTHeWy8YnJNk9ydFpI5zNvF40y3IeBry6v14OPIN2V/xU2p3jubwf+B7rd1k7\nDDivqs6c/0eZ6NUTXofOTExyd+CpwFlV9dbRGavqJ7RgaxvaRfPN6dXvjI/lv54W3O0A/Oomlnu+\nvlJVx09IPwjYAzhmfBtWaz15M7ACWLUIZbiwqt48lnbCyP9/OjbtZFog8cuzLO/4qvrPsbRXA9+n\ntfjNjNb3UGBv4HNV9f7RzFX1T8BZwH17a8+4YyesY4M2se5f0PepmXnW0lqStgfuNZLvebR97uiq\nusWIfFU1eqweTgtoX1RV3xnL9ylaa82Tktx+ZFKA/52w3B9V1eBH0JO0OOyuJmlrtgft4nKmq1SA\ni2ldrsYdXSMP+SfZBXgS8P9oF1krxy7OblZVP0lyMnB4f45iT9od9T/e1A9QVbcYvnrMStoFZdIG\nYBh3u/53v9HEJL9A61b1EFrLyG1HV0sLILaEr8yS/qBejp+f5XPtS6vP/WjdqDbF1yekzXRD+2ZV\n/Xh0QlXdmOS7wD0mzFe07ovrJ1Zdm+Qc4NdpZT8XmHmu6LOzlOszwAOBX6EFPKPrmG27bdBG1v3V\nVXXJhPSZtO1H0h7Y/54+j+LMBHCPTPLrE6bvQLsW2ZvWnfBDtNasv0vyBFp3xi/ULQf9kKQ5GeRI\nmnZX0rq87Do+oar+lT6ASpJtaF1fZrPeqFT9bvdx/RmAVwCvBJ47x/x/D7yANpz0nrQ7zSfN+1Ns\nvLv1vw9k3cXluALuOPMmyUOAT9I+86dp3ZC+T2uduD/wm6x/4bs5XTlL+t1o5bvFwAkjivYsyqaa\n9AzHjXNMm5l+61mmrZ0lfeaz3mXkbwG3aFnprqBtg7vOsawF2YS6v3aWRc5sp9FgfKa8E28KjJnZ\nf182R56b67mqLkqyktZtc3/as1NJ8m3gTVX1N/NYpyQZ5Eiael8AHkF7xuP4OfJt7NC6X+p/V86V\nqar+M8lZtG5qdwHeVyMPmG9GMxfhb6qqI+Y5z6toF7K/Md4VLMkraRe6CzEzmMNs3xmTLtJnzDYg\nwXV92uOr6hMLLM9S22mW9JmuldeN/M1I+rhdaNtgUqC1sb9btNh1P8m1wO60FqFvbSDvzGe7w2hX\nuLn0Vpun9hsXvwQ8lnaD4W1Jrq+qLXFzQdJWzmdyJE2742l3k5+cZN/NsPyZbjjzOR/+PW30qVvT\nRgrbEr5Eu+B96ALm2Qv4zizPC62aZZ6fsf7d+lEzwdxu4xOS3JnW1WihzqIFALcYqnnKBXj4LRLb\nCHj3ow1KMDNS30w3uVWzLOuR/e98R7GDVk8we11tTN0v1EzXugMWkHfB9VxVN1XV16vqjcDTadv+\nSQtdjqTlySBH0lSrqm8Bf067O316kgfPknX7WdJhlrviSW5LezC6mP25iVGn0C6yDupd5Ta7qrqy\nr/dBSY7od7fXk/Z7MPccSboYuHuS8ed0nsO6C+txVwM7JrlFF62qupY2RPDDktz8AHp/wP5tbFzX\ntw/0cr5wjt9veXCS22zEsje3Q5PcbyzttbSBLE6uqplA5HO07bYqyUGjmZM8lfa8yrk1y28/zeKa\n/vees0y/mIXX/UIdS2vdO3rSjYeM/A4S8Ne0wOytSfaakPfWvYvdzPtfTbLdhHXOtIb9cJNKLmnZ\nsLuapKlXVa9JAq0rzheSfBX4Mu2C7660AQgeTQtWJgUfAR6RvpD+fhfanegVwIXA6+ZRjh+z/m+T\nbCnPpd2h/wvgmUk+z7rfP7kPbbSsp9CGXIb2OymPAs5MciptOOKVtIvq9wNPnrCOT9NGE/tEkn+j\n/f7J16vqtD79TcDfAmcleV+f/gjatvwGYwMfbEhV/TTJbwEfpwWvXwDOBn5Mu4B/AK1e7866C/tp\nULSH4b/Yt+2VtJadB9MCmlfcnLGqkhza8/9zkg/SWnn2o/1uzbW0Uf7mv/Kq65P8O21/Pok2RPVN\nwAeq6lw2ru4XpHfdfAEtgDk7yYdox9DdaPV2Na2LGVV1bpLDaK2g5yb5OHABLTC+J62F8jJaKxi0\n4bsP6/vghbRttDetm92PaUG1JG2QQY6krUIPdP4J+CPaxfUhtIftv0+7GPob2l30syfNTusuM9pl\n5ke0i9J3A2/pQ+xOmm/eRVxg/nkvv1/YPpT2I4uHAL9Nu0hcS7tgfBFtpK6Z/KclOZB2wf07tO5+\nX6J1V9qvzz/uz2gtEU8EfoPWHeqdtCF+qaq/T1K0EeUOpXVh+0Bfx0dm+SxzbpOqOru3iLy4r/dZ\ntAv2K2jDer+CdV3l5mOhZdhQnc027U20UcBeRPtRy+/Tui++oqrWC8iq6swkD6ANbPFI2sX6VbRh\nql9bVRfOsf7ZPI02KuABtP0hwEW0VqGNqfu5PuvkzFV/m+Q/gJf2ZT+J9rn+A3jHWN6TkpxNq+dV\ntAEFfkgb4e6faMO4zziZtu/9Oi14vz0tCDqZdpxu8AeBJQn6j3ttMFNyMe3hwZuAG6pqZZLtab/2\nvDutefzgqrqu5z+S9ivlN9LGxj+jp9+f1r/+dsBpVbXJw69KkrQlJHkt7TeWHlpVX1zq8kiSZjff\nZ3JuAlZV1a9U1cwIREcAn6qqfWl3EI8ESHIf2pCg+9HuMh070kXkOOCwqtoH2CfJ/ov0OSRJkiQJ\nmH+Qkwl5D2LdL0afwLoRTw4ETqmqG6vqYlpXipVJdga2q6qZHzg7EUdJkSRJkrTI5hvkFPDJJF9J\n8vs9baeqWgs3j/6zY09fwbpfSIbWl3ZFf106kn4pW+4XtyVJkiQtE/MdeOAhVXVFkrsDZyT5Jrd8\nSHFjf7hMkqSpV1Wvoo3wJ0macvMKcqrqiv73u30IzJXA2iQ7VdXa3hXtOz37Zaz/g3H36Gmzpd9C\nH8FHkiRJkmZVVZmUvsHuaknukORO/f870sa+/wbttyKe2bMdShtOk57+1CS3SbInbXz7L/cubdcl\nWdkHInjGyDyTCjx1r6OOOmrJy7DcX9bBdLysh+l4WQ9L/7IOpuNlPUzHy3pY+tdyq4O5zKclZyfg\nA711ZVvgPVV1Rv8xslOTPBtYQxtRjWo//HUqcC5wA/C8WleKw1l/COnT57F+SZIkSZq3DQY5VXUR\n7Vewx9Ovof3C+KR5Xg+8fkL6V4FfXHgxJUmSJGl+5ju6moBVq1YtdRGWPetgOlgP08F6WHrWwXSw\nHqaD9bD0rIN1sqH+bEshSU1juSRJkiRNhyTUxg48IEmSJElbE4McSZIkSYNikCNJkiRpUAxyJEmS\nJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDsu1SF2A2SZa6\nCJvdTrvvzpUXX7zUxZAkSZIGJVW11GW4hSTFFJZr0SVM4/aXJEmSpl3atfTElhG7q0mSJEkaFIMc\nSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJ\nGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjk\nSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ\n0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUg\nR5IkSdKgGORIkiRJGpR5BzlJtknytSQf7u+3T3JGkm8m+USSu4zkPTLJBUnOS/LYkfT7JzknyflJ\njlncjyJJkiRJC2vJeRFw7sj7I4BPVdW+wGeAIwGS3Ac4GNgPOAA4Nkn6PMcBh1XVPsA+SfbfxPJL\nkiRJ0nrmFeQkuQfweOAfRpIPAk7o/58APKn/fyBwSlXdWFUXAxcAK5PsDGxXVV/p+U4cmUeSJEmS\nFsV8W3L+CvgToEbSdqqqtQBVdSWwY09fAVwyku+ynrYCuHQk/dKeJkmSJEmLZoNBTpInAGur6mwg\nc2StOaZJkiRJ0hax7TzyPAQ4MMnjgdsD2yU5CbgyyU5VtbZ3RftOz38ZsNvI/PfoabOlT3b00ev+\nX7WqvSRJkiQtS6tXr2b16tXzypuq+TfAJHk48JKqOjDJXwJXV9Ubk/wpsH1VHdEHHngP8EBad7RP\nAveqqkpyFvBC4CvAx4C3VdXpE9ZTLKBcW62EhWx/SZIkSU3atfTEnmbzacmZzRuAU5M8G1hDG1GN\nqjo3yam0kdhuAJ5X667kDweOB24HnDYpwJEkSZKkTbGglpwtxZYcSZIkSXOZqyVnIb+TI0mSJElT\nzyBHkiRJ0qBsyjM5GrCd99iDtWvWLHUxtoiddt+dKy++eKmLIUmSpEXiMzlLaYqfyUnCsqgDmOp6\nkCRJ0mQ+kyNJkiRp2TDIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAjSZIk\naVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQ\nI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIk\nSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSD\nHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIk\nSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGZYNBTpLbJvlS\nkq8n+a8kr+vp2yc5I8k3k3wiyV1G5jkyyQVJzkvy2JH0+yc5J8n5SY7ZPB9JkiRJ0nK2wSCnqn4C\nPKKqfgW4H/DIJA8BjgA+VVX7Ap8BjgRIch/gYGA/4ADg2CTpizsOOKyq9gH2SbL/Yn8gSZIkScvb\nvLqrVdWP+r+37fN8DzgIOKGnnwA8qf9/IHBKVd1YVRcDFwArk+wMbFdVX+n5ThyZR5IkSZIWxbyC\nnCTbJPk6cCWwuqrOBXaqqrUAVXUlsGPPvgK4ZGT2y3raCuDSkfRLe5okSZIkLZpt55Opqm4CfiXJ\nnYFPJFkF1Hi2RS3Z0Uev+3/VqvaSJEmStCytXr2a1atXzytvqhYWmyR5FfBj4DBgVVWt7V3RPltV\n+yU5AqiqemPPfzpwFLBmJk9Pfyrw8Kp67oR1FAss11YpYaHbf0tJwrKoA5jqepAkSdJkaddwmTRt\nPqOr7TAzclqS2wOPAb4OfBh4Zs92KPCh/v+HgacmuU2SPYG9gS/3Lm3XJVnZByJ4xsg8kibYeY89\nSLIsXjvvscdSb25JkjQQ8+mutgtwQg9MtgFOqqpP92d0Tk3ybForzcEAVXVuklOBc4EbgOfVutvk\nhwPHA7cDTquq0xf100gDs3bNmmXTorY2E2/ESJIkLdiCu6ttCXZXW3p2V5sO1oMkSdJkm9RdTZIk\nSZK2JgY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRB\nMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4k\nSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQN\nikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIk\nSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRp\nUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAj\nSZIkaVAMciRJkiQNygaDnCT3SPKZJP+V5BtJXtjTt09yRpJvJvlEkruMzHNkkguSnJfksSPp909y\nTpLzkxyzeT6SJEmSpOVsPi05NwIvrqr7Ag8GDk9yb+AI4FNVtS/wGeBIgCT3AQ4G9gMOAI5Nkr6s\n44DDqmofYJ8k+y/qp5EkSZK07G0wyKmqK6vq7P7/D4DzgHsABwEn9GwnAE/q/x8InFJVN1bVxcAF\nwMokOwPbVdVXer4TR+aRJEmSpEWxoGdykuwB/DJwFrBTVa2FFggBO/ZsK4BLRma7rKetAC4dSb+0\np0mSJEnSotl2vhmT3Al4P/CiqvpBkhrLMv5+0xx99Lr/V61qL0mSJEnL0urVq1m9evW88qZqw7FJ\nkm2BjwIfr6q39rTzgFVVtbZ3RftsVe2X5AigquqNPd/pwFHAmpk8Pf2pwMOr6rkT1lfMo1xbvYT5\nbP+lkIRlUQdgPUyLKa4HSZI0fdKuHTJp2ny7q70LOHcmwOk+DDyz/38o8KGR9KcmuU2SPYG9gS/3\nLm3XJVnZByJ4xsg8kiRJkrQoNtiSk+QhwOeAb9C6pBXwcuDLwKnAbrRWmoOr6to+z5HAYcANtO5t\nZ/T0XwWOB24HnFZVL5plnbbkLDFbEKaD9SBJkjTZXC058+qutqUZ5Cw9L66ng/UgSZI02WJ0V5Mk\nSZKkrYJBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJkiRp\nUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmDYpAj\nSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJ\ng2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMc\nSZIkSYNikCNJkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJ\nGhSDHEmSJEmDYpAjSZIkaVAMciRJkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjk\nSJIkSRqUDQY5Sd6ZZG2Sc0bStk9yRpJvJvlEkruMTDsyyQVJzkvy2JH0+yc5J8n5SY5Z/I8iSZIk\nSfNryXk3sP9Y2hHAp6pqX+AzwJEASe4DHAzsBxwAHJskfZ7jgMOqah9gnyTjy5QkSZKkTbbBIKeq\nPg98byz5IOCE/v8JwJP6/wcCp1TVjVV1MXABsDLJzsB2VfWVnu/EkXkkSZIkadFs7DM5O1bVWoCq\nuhLYsaevAC4ZyXdZT1sBXDqSfmlPkyRJkqRFtVgDD9QiLUeSJEmSNsm2Gznf2iQ7VdXa3hXtOz39\nMmC3kXz36Gmzpc/u6KPX/b9qVXtJkiRJWpZWr17N6tWr55U3VRtuhEmyB/CRqvrF/v6NwDVV9cYk\nfwpsX1VH9IEH3gM8kNYd7ZPAvaqqkpwFvBD4CvAx4G1Vdfos6yvmUa6tXsJ8tv9SSMKyqAOwHqbF\nFNeDJEmaPmnXDpk0bYMtOUn+EVgF3C3Jt4GjgDcA70vybGANbUQ1qurcJKcC5wI3AM+rdVcthwPH\nA7cDTpstwJEkSZKkTTGvlpwtzZacpWcLwnSwHiRJkiabqyVnsQYekCRJkqSpYJAjSZIkaVAMciRJ\nkiQNikGOJEmSpEExyJEkSZI0KAY5kiRJkgbFIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQ\nDHIkSZIkDYpBjiRJkqRBMciRJEmSNCgGOZIkSZIGxSBHkiRJ0qAY5EiSJEkaFIMcSZIkSYNikCNJ\nkiRpUAxyJEmSJA2KQY4kSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKgGORIkiRJGhSDHEmSJEmD\nYpAjSZIkaVC2XeoCSNK023mPPVi7Zs1SF2Oz22n33bny4ouXuhiSJG2yVNVSl+EWkhRTWK5FlzCN\n2x8gCcuiDsB6mBbWw9Kb4jqQJGlc2vdWJk2zu5okSZKkQTHIkSRJkjQoBjmSJEmSBsUgR5IkSdKg\nGORIkiRJGhSHkJYkTb3lMow3OJS3JC0Gh5BeSlM8XOuyGTIXrIdpYT0sPetgOkxxPUjSNHEIaUmS\nJEnLht3VJEnSvNhtUNLWwu5qS2mKuyTYNWQ6WA/TYdnUg3UwHayH6TDF9SCpsbuaJEmSpGXDIEeS\nJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQDHIkSZIkDYpBjiRJkqRBMciRJEmSNCjbLnUBJEmS\nNH8777EHa9esWepibHY77b47V1588VIXQ1upbOlf803yOOAYWivSO6vqjRPy1LL4ReUp/jVlf9V6\nOlgP02HZ1IN1MB2sh+lgPSy9Ka4DTYe0fSSTpm3R7mpJtgHeDuwP3Bc4JMm9t2QZNsnq1UtdAlkH\n08F6mA7Ww9KzDqaD9TAdrIclt9o6uNmWfiZnJXBBVa2pqhuAU4CDtnAZNp47ztKzDqaD9TAdrIel\nZx1MB+thOlgPS84gZ50tHeSsAC4ZeX9pT5MkSZKkReHoapIkSZIGZYsOPJDkQcDRVfW4/v4IoMYH\nH0jiU2aSJEmS5jTbwANbOsi5FfBN4FHAFcCXgUOq6rwtVghJkiRJg7ZFfyenqn6W5PnAGawbQtoA\nR5IkSdKi2eK/kyNJkiRJm9MgBx5IcmSSdyx2XklLL8lxSV6x1OVYLElOS/J7S12OaZfkdkk+kuTa\nJO/dwuv+bJJnb8l1LrYkuye5qf9e3YL2uyF8/q3V0M53Whz9WP75pS7HtNsqgpwkz0xyTpIfJrk8\nybFJ7jJb/qp6fVX94XyWvZC8mizJSUneNZb28CRXJdlpqcq13CS5OMmPklyX5Jokn0/ynCQTH8hb\nwHIvSvLIRSznLU7OSY5KctJ85q+q51bVXyxWeeaSZGWSjyX5Xt+fz0ryzMVcR1U9vqrm9dmX0uba\nv0aW/ye+YTggAAAPIUlEQVRJzu/n+YuTvC7JbUayPBm4O7A98LQk30/ygJH5f7fvW+Npy65L9Ehd\nXd+30/XArsDNXTcWa78bD5562qE97aVjeS9J8rBNXeeQbOi4mu/5brHP05q/Xof/m+TnxtK/3o+D\ne27i8ifdZLAb1jxMfZCT5CXA64GXAHcGHgTsDnwyyS2eKeqDG2jLehHwuCSPAkhyW+AdwP+tqrWL\ntZLRL1FNVMATquoutGPkDcCfAu9c0lLd0mwn56k6aSd5MPBp4LPAXlW1A/BcYP+NXN4tgoFNCRCW\n4Fy32favJH8N/D7wdGA74ADaADWnjmTbHTi/mp8BXwRGL5gfCpw3Ie1fN7V8W6GZurpzVW1XVXcG\nLt9M60pf3/i+fA3wsiR33EzrHYqt5byt2RVwEXDITEKSXwBuz+b7XluUm0tDN9UXjUm2A44Gnl9V\nn6yqn1XVt4GDaSeDp/c7wO/rrQnXAoeO3xVO8oweaX83yStH73iM5h25I/WMJGuSfCfJy7f4B9/K\nVNU1wAuBdyS5A63O/qeqTkrz8iT/07fnP860wvVp70tyRb+D9Zkk955Zbq/Ttyf5eJLvA7+xFJ9v\nKzNz9+/7VfVR4Hdox8R9ktwmyZv7vn1FWovobQGS3C2tK9D3klyd5F97+onAPYGP9LvCL51txQst\n46wTWyvgJUlenGRtkssy0nqS5N1JXjPy/k/SWngvTfKsjLQUjd8B63eY/23k/b2TnNE/83lJnjJS\nlL8E3l1Vb+77OFX19ao6pM97177NvtPn/0iSFSPL/mySP+93Zn8I7DlH2mgZn53k3L7Mj4/eBeyf\n7XlJzgfOn/8mXzRz7V+PT/K1tDvSa5IcNVLujyY5fL0FJf+R5KAke9OCx6dV1Zer6qY+IM1vA/sn\nWZXkaODVwFP7fvgs4N+4ZUDzRuDhY2mf6+tLkiP6uei7SU5JcteR8jwoyRf6MfD1JKPLGS33Lr3s\nL9m4TbjFbOg4u3m/S7JNkrf07XJhksMz1joD7NH32+uTnJ51d61ngshr+7QH9vfnAWfSblBOWv/4\ncfzwJJeMvL8oyUvTenFcn+QfkuyY1s3uun7cztqjYysz6bh6Rj+ubt5OCz1PJzk17Vz/vSSrk9zn\n5hW25b69H5vXJzkzyZ4j0+87cm68Iu0nPzZ4HC1jJwGHjrw/FDhh5k2SOyc5sX9fXJSRLojp30tJ\n3pR2LXRhkv37tD+nncfe3uvpbSPreExa6/c1Sd6+eT/e1mmqgxzg14HbAh8YTayqHwIfBx7Tkw4E\nTq2quwL/OJMNoB/Uf0OLsHcB7kJrtl9vkWPvHwLcC3g08Ook+y7Ghxmyqno/8DXgn2h3ZP+gT3ox\n7a7sbwD3AH4A/PXIrB8B9gJ2Bv6TdqIYdQhwVFVtR/vC1AJU1VeAS2knyTcAewP3639X0C4coV2I\nXALcDdgReHmf/xnAt4En9rvCb95CRd+Zdkd/V9r+9DeTLmiSPI62jz2Kdcfshu6czZwb7kAb6fFk\nYAfgqcCxaYHP7YEHA/88x3K2Ad4F7Ea7wPgRMP5F8/Re/u1o23G2tJnPcxBwBPAkWtesf6MdU6MO\nAh4A3IclNrZ//QD4vX5H+gnAHyU5sGc9Abj5+Y8kv0Sr24/R6u6Sqvrq2LIvBb4EPKaqjgZeB5zS\n98N304KXh/Tl7QDcgdbys3Ikbb+eD9qNmAN7WXcFvgcc2/OuAD4KvKaqtgdeCvxzkruNlinJHsBq\n4G1V9ZaN2GTT6g9pLZT3A+5P2//Gj6NDaBdud6d9L8/c8JgJNO/c6+ZL/X0BrwL+eAEXwePr/C3g\nkcC+wG/SvveP6GW4Fa1OB6cfV5fR9tVRCz1Pn0b7ft2R9v38nrHl/Q5wFHBX4ELgLwCS3An4ZJ9/\nF9r3xaf7PLMeR8vcWcB2SfbtNwd+h/bdMuPttHP+HsAqWhD7rJHpK2k3Bu4GvIn23UJVvZL2PfD8\nXrej+/wTgF8Ffgk4OMljN8Pn2qpNe5CzA3BVVd00YdoVfTrAmVX1EYCq+t+xfL8NfLiqzqyqG1l3\nUTebov1g6U+r6hzgP2g7kDbscNoX0p9V1UzXiOcAL6+qK6vqp8BrgadA+xXYqjqxqn7Up70G+NV+\ngTnjA1X15Z7/hi32SYblctqJ8w9pXQiv6zcK3sC65vUbaF9me/YW0y+MLWNLN43/FHhtL8vHaRfQ\nk242PIXW2nJeVf2Y1oo4X08ELur7YFXVf9CCmqfQnvvYhnaemaiqrqmqD1TVT/r2fD3rtywAHF9V\n/91bJ26cI23Gc4DXV9X5/bz3BuCXk+w2kud1vQ5/soDPujldDvxcVX2uqv4LoKr+EziFda0qHwbu\nlWSv/v7pwHv759+B2bfz6Hl+3JeAOyT5RdpNlM/38/+3RtIu6sEStG37iqq6op9LXgM8uV+Q/C7w\nsar6RC//p4F/Bx4/sr770rouvqqqtoauRB/sd3ivSfIvG8j7FOCtfdtcR9vvxr27qi7s+92pwC+P\nTb/FOaJ/h36S1v1qY/x1VV1VVVfQLvTOqqpz+vfFB4Bf2cjlbg0uB35uLG1B5+mqOr5/v87s77+U\n1kNmxgeq6qv9XPMe1tXpbwJXVNUx/Vrohz3wgrmPo+VupjXnMbSAZeY6aFta0HNEr481wFsYufED\nrKmqd1VV0W4K7ZJkxw2s7/W99e8S2rlp/Jhc9qZ9p7wK2GGWg2eXPh3anY3Z7Do6vV8IXb2B9Y4+\nR/Ij4E4bLqqq6ju0Ojl3JHmmCf2aJNcA5wA39W4H2yT5y940ey1wAS3IHL2omatuNT8raHc97wB8\ndaQuPk4LfqDdOboQOKN3Q9jYi5L5+Blw67G0W9O+wGdcPXZzY7bjcL3jG1jD/AOy3YEHjVwIfg94\nGrAT7e7kTbTzzERJbp/k79K6wl5L67Zz12S952wm7b9z7dO7A28dqaOracfEipE8l06cc+msAK5J\nG6ThM707xrW0i6EdAEYujJ/et88hwIl9/quYfTuPnufX05f5ZVog9TDaRTDAF0bSPjcyy+7AB0a2\n7bm0fW6nPu3gsX3hIbQWxRlPo237uVr3pslBVfVz/fVbG8g7fhxN2kevHPl/Id+LrwaeO48LtklG\nv4t/POH9kL+bV9Ceaxo17/N0/359Q893Le2ZkfHv19nq9B59PZPMdRwtdyfTzhPPZN35Ddo235b1\nW+7XsP55/ea66NepsOH922vVDZj2IOdM4Ce0Juub9abUA1jXfDpX95QraAfszLy3Z92FnTa/S2jd\nTWa+bLevqjv2gOgZwOOAVdW6Gu5Nu0AdvUicqofRtzZpI03tCnyQdhK870hd3LV3LaKqflBVL62q\nvWhdEV6c5BF9MYtdB9+mNdmP2pN20l+oK2jdxWbszvrl/SEtuJsxetF6CbB6bN+8c1U9v3/JnElr\nCZ7NS2hd5B7Q99+ZVpwN7b9zbc9LgOeMlelOVXXWPOffokb2r8/Tugp/EFjRt8ffsf62OJHWgvMo\n4IczLbTAZ4Ddkvza2LJ3ow0086k5ijDzXM5vsC7ImUm7+Xmc7tvAARPORVfQtvuJY9O2q6o3jcx/\nNC3g+qexQHZaLaSM631P0m5Ozdec+2NVfRP4F+AVzH1sznpDYbkZO65utsDz9NNoLTKP7MfjHtzy\n+3U2l9C6uU0y13G0rFV7Zvwi2vXpaOvpd2mB4O4jabvTuiTOa9GLUsBlaKqDnKq6ntYU+tdJ9k+y\nbe8T/V7agTaf4S/fD/xm2kOlt2bD3Vm2hi+vrcnfAa+f6W7TW3B+s0/bjhbEfi9tBJ7X4cG8KJJs\nl+SJtOc5TqqqbwD/AByT5O49z4qZPrxJnjDSlej7wI20Fhdod4sWczz+9wKv7OtPkkfTuo69fyOW\ndSrwzCT79Wdsxrujng38Vm912Rs4bGTaR4F9kjy9n1tuneTXsm7wi5f1Zb8k/SHrJL+UZOYZme1o\nd5Ov79OP3ojyj/tb4OX9WUKS3CXJkxdhuYtqwv71X7S7iN+rqhuSrKRdZN2sqs6kHd9vYeTcXVUX\n0M4T70nywH4H+r60/eGMqvrsHEX5HPAIYLdqgxVAa8lZRetmPBrk/B3wuvSBHJLcfeSZoZNp3xOP\n7eu/XdqD8KPPb95A69Z1R+CkrSTQGTdbmU8FXpRk1/78zMsWsMzv0lo9Z7sohvY9/izasx8zzgYe\nn2T7JDvTRulc1mY5rkanL+Q8Pf79+nrm//36UWDnJC9MG7DmTv2YhrmPI8GzaYHlTGtMaHX0PuAv\n+rbcHfi/zO8aFhb/O3jZmOogB6DfSXs58GbgOtrd1TXAo2sez2hU1bnAC2gXVpcD1wPfoR38E2fZ\nwHvNbXx7vYXWLerTSa6j3ZmauWP7btodxMuBbzB212rCsrRhH+nb+dvAkbTjZmbkrpcB/wOc1bsv\nnAHs06fdC/hU2ih2XwD+pqpmLhBfD7yqd0948SKU8TW04X8/T+uO8QbayFrnzjHPxH2hqk4HjqG1\nBpzPutbdGX9Fuzi9kra/nTwy7w+Ax9IGHLi8v94A3KZPP5P2jNmjgAuTXEULQj7WF3EM7U70Vf3z\nnDaPMs+ZVlUf7GU4pdfRObTWzrnm35Lm2r+eB7y2T38l7Zw77kTgF1j/gVyq6nBaEH4y7eLtNFqd\nbijA+yLtpwVubumqqqtpF95rq2q0y81bgQ/Ruvpc1+dd2ee5lDagw8v7vGtoD9bPfEdWz3cjrWfB\njkz3EL/zGaZ99P+/p50PzgG+StvHbxzpMjrrftcv5v4C+EI/R6yckOdi2gXd6HDSJ/X1XQycTnuG\na67PsNT7/uY013E1aiHn6RP68i6jDerzxfkWpp8bH0NrLbqSdm5d1SfPehwtY6Pn8Iuq6msTpr2A\n1pviW7SbLydXG0Blg8ukbfOnpI10d8yE6ZPeC0h7xmn56Hc0rgX27g9/SRqQJDfRju9vLXVZtL4k\nTwf+sKr8QcgpljZq4XFVtecGM0vSlJr6lpzFkOSJvbvKHWktC+cY4EjSltO7Ex5O6+6iKdK75x2Q\n5FZpw2kfxfrPFEjSVmdZBDm0bgiX00bG2YvWPUXSMC2v5umtQH/26zu07qnjv/ujpRfgz2jdR78K\n/Bct0JGkrday664mSZIkadiWS0uOJEmSpGXCIEeSJEnSoBjkSJIkSRoUgxxJkiRJg2KQI0mSJGlQ\nDHIkSZIkDcr/B901TsoObwv2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ebffbe0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot GBM Feature Importances\n",
    "importances = data_gbm2.varimp(use_pandas=True)\n",
    "feature_labels = list(importances['variable'])\n",
    "Index = importances.index\n",
    "\n",
    "plt.figure(figsize=(14,5))\n",
    "h = plt.bar(range(len(feature_labels)), importances['relative_importance'],width=0.6, label=feature_labels, color ='aqua')\n",
    "plt.title(\"GBM Feature Importances\", fontsize=20 )\n",
    "xticks_pos = [0.65*patch.get_width() + patch.get_xy()[0] for patch in h]\n",
    "plt.xticks(xticks_pos, feature_labels, fontsize=12,  ha='right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>relative_importance</th>\n",
       "      <th>scaled_importance</th>\n",
       "      <th>percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Origin.MDW</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.008451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Year.2003</td>\n",
       "      <td>0.866093</td>\n",
       "      <td>0.866093</td>\n",
       "      <td>0.007320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dest.EWR</td>\n",
       "      <td>0.774157</td>\n",
       "      <td>0.774157</td>\n",
       "      <td>0.006543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Origin.ORD</td>\n",
       "      <td>0.742141</td>\n",
       "      <td>0.742141</td>\n",
       "      <td>0.006272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Year.2001</td>\n",
       "      <td>0.736653</td>\n",
       "      <td>0.736653</td>\n",
       "      <td>0.006226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Year.2002</td>\n",
       "      <td>0.710980</td>\n",
       "      <td>0.710980</td>\n",
       "      <td>0.006009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Origin.SFO</td>\n",
       "      <td>0.678087</td>\n",
       "      <td>0.678087</td>\n",
       "      <td>0.005731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Month.1</td>\n",
       "      <td>0.666030</td>\n",
       "      <td>0.666030</td>\n",
       "      <td>0.005629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Origin.AUS</td>\n",
       "      <td>0.665489</td>\n",
       "      <td>0.665489</td>\n",
       "      <td>0.005624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Dest.SFO</td>\n",
       "      <td>0.657573</td>\n",
       "      <td>0.657573</td>\n",
       "      <td>0.005557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Year.1989</td>\n",
       "      <td>0.640961</td>\n",
       "      <td>0.640961</td>\n",
       "      <td>0.005417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Year.1990</td>\n",
       "      <td>0.637166</td>\n",
       "      <td>0.637166</td>\n",
       "      <td>0.005385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Year.2007</td>\n",
       "      <td>0.604424</td>\n",
       "      <td>0.604424</td>\n",
       "      <td>0.005108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Year.1994</td>\n",
       "      <td>0.591667</td>\n",
       "      <td>0.591667</td>\n",
       "      <td>0.005000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Origin.PSP</td>\n",
       "      <td>0.591646</td>\n",
       "      <td>0.591646</td>\n",
       "      <td>0.005000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Dest.FLL</td>\n",
       "      <td>0.591537</td>\n",
       "      <td>0.591537</td>\n",
       "      <td>0.004999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Year.2008</td>\n",
       "      <td>0.587823</td>\n",
       "      <td>0.587823</td>\n",
       "      <td>0.004968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Dest.CAE</td>\n",
       "      <td>0.585480</td>\n",
       "      <td>0.585480</td>\n",
       "      <td>0.004948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Origin.ATL</td>\n",
       "      <td>0.567194</td>\n",
       "      <td>0.567194</td>\n",
       "      <td>0.004794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Origin.PIT</td>\n",
       "      <td>0.558928</td>\n",
       "      <td>0.558928</td>\n",
       "      <td>0.004724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Distance</td>\n",
       "      <td>0.555028</td>\n",
       "      <td>0.555028</td>\n",
       "      <td>0.004691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Year.1998</td>\n",
       "      <td>0.552688</td>\n",
       "      <td>0.552688</td>\n",
       "      <td>0.004671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dest.CHO</td>\n",
       "      <td>0.549932</td>\n",
       "      <td>0.549932</td>\n",
       "      <td>0.004648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>FlightNum</td>\n",
       "      <td>0.549580</td>\n",
       "      <td>0.549580</td>\n",
       "      <td>0.004645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Origin.IAH</td>\n",
       "      <td>0.549099</td>\n",
       "      <td>0.549099</td>\n",
       "      <td>0.004641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Dest.PHX</td>\n",
       "      <td>0.543288</td>\n",
       "      <td>0.543288</td>\n",
       "      <td>0.004591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>UniqueCarrier.HP</td>\n",
       "      <td>0.537485</td>\n",
       "      <td>0.537485</td>\n",
       "      <td>0.004542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Origin.PHL</td>\n",
       "      <td>0.533020</td>\n",
       "      <td>0.533020</td>\n",
       "      <td>0.004505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Dest.HTS</td>\n",
       "      <td>0.524996</td>\n",
       "      <td>0.524996</td>\n",
       "      <td>0.004437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>UniqueCarrier.UA</td>\n",
       "      <td>0.524380</td>\n",
       "      <td>0.524380</td>\n",
       "      <td>0.004432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>Dest.GSP</td>\n",
       "      <td>0.256756</td>\n",
       "      <td>0.256756</td>\n",
       "      <td>0.002170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286</th>\n",
       "      <td>DayOfWeek.4</td>\n",
       "      <td>0.256518</td>\n",
       "      <td>0.256518</td>\n",
       "      <td>0.002168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>287</th>\n",
       "      <td>Origin.CRP</td>\n",
       "      <td>0.255690</td>\n",
       "      <td>0.255690</td>\n",
       "      <td>0.002161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>Dest.GRR</td>\n",
       "      <td>0.254057</td>\n",
       "      <td>0.254057</td>\n",
       "      <td>0.002147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>Dest.HPN</td>\n",
       "      <td>0.252856</td>\n",
       "      <td>0.252856</td>\n",
       "      <td>0.002137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>Origin.LAN</td>\n",
       "      <td>0.251714</td>\n",
       "      <td>0.251714</td>\n",
       "      <td>0.002127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>291</th>\n",
       "      <td>Origin.TPA</td>\n",
       "      <td>0.249546</td>\n",
       "      <td>0.249546</td>\n",
       "      <td>0.002109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Origin.CLT</td>\n",
       "      <td>0.248593</td>\n",
       "      <td>0.248593</td>\n",
       "      <td>0.002101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>293</th>\n",
       "      <td>Origin.CVG</td>\n",
       "      <td>0.248328</td>\n",
       "      <td>0.248328</td>\n",
       "      <td>0.002099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>Origin.PVD</td>\n",
       "      <td>0.243274</td>\n",
       "      <td>0.243274</td>\n",
       "      <td>0.002056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>295</th>\n",
       "      <td>Dest.RSW</td>\n",
       "      <td>0.238737</td>\n",
       "      <td>0.238737</td>\n",
       "      <td>0.002018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>Dest.MAF</td>\n",
       "      <td>0.237673</td>\n",
       "      <td>0.237673</td>\n",
       "      <td>0.002009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>Dest.CAK</td>\n",
       "      <td>0.231845</td>\n",
       "      <td>0.231845</td>\n",
       "      <td>0.001959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>Origin.BDL</td>\n",
       "      <td>0.231663</td>\n",
       "      <td>0.231663</td>\n",
       "      <td>0.001958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>Origin.SNA</td>\n",
       "      <td>0.228553</td>\n",
       "      <td>0.228553</td>\n",
       "      <td>0.001932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>Origin.ONT</td>\n",
       "      <td>0.225977</td>\n",
       "      <td>0.225977</td>\n",
       "      <td>0.001910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>Dest.SAN</td>\n",
       "      <td>0.224082</td>\n",
       "      <td>0.224082</td>\n",
       "      <td>0.001894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>Origin.STX</td>\n",
       "      <td>0.223197</td>\n",
       "      <td>0.223197</td>\n",
       "      <td>0.001886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>Dest.ANC</td>\n",
       "      <td>0.215848</td>\n",
       "      <td>0.215848</td>\n",
       "      <td>0.001824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>Origin.TYS</td>\n",
       "      <td>0.214950</td>\n",
       "      <td>0.214950</td>\n",
       "      <td>0.001817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>305</th>\n",
       "      <td>Origin.ELP</td>\n",
       "      <td>0.214924</td>\n",
       "      <td>0.214924</td>\n",
       "      <td>0.001816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>306</th>\n",
       "      <td>Origin.SRQ</td>\n",
       "      <td>0.203653</td>\n",
       "      <td>0.203653</td>\n",
       "      <td>0.001721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>Dest.MKE</td>\n",
       "      <td>0.194508</td>\n",
       "      <td>0.194508</td>\n",
       "      <td>0.001644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>Origin.EYW</td>\n",
       "      <td>0.179053</td>\n",
       "      <td>0.179053</td>\n",
       "      <td>0.001513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309</th>\n",
       "      <td>Dest.missing(NA)</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>310</th>\n",
       "      <td>Origin.missing(NA)</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>311</th>\n",
       "      <td>Year.missing(NA)</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>312</th>\n",
       "      <td>UniqueCarrier.missing(NA)</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>DayOfWeek.missing(NA)</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>Month.missing(NA)</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>315 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      variable  relative_importance  scaled_importance  \\\n",
       "0                   Origin.MDW             1.000000           1.000000   \n",
       "1                    Year.2003             0.866093           0.866093   \n",
       "2                     Dest.EWR             0.774157           0.774157   \n",
       "3                   Origin.ORD             0.742141           0.742141   \n",
       "4                    Year.2001             0.736653           0.736653   \n",
       "5                    Year.2002             0.710980           0.710980   \n",
       "6                   Origin.SFO             0.678087           0.678087   \n",
       "7                      Month.1             0.666030           0.666030   \n",
       "8                   Origin.AUS             0.665489           0.665489   \n",
       "9                     Dest.SFO             0.657573           0.657573   \n",
       "10                   Year.1989             0.640961           0.640961   \n",
       "11                   Year.1990             0.637166           0.637166   \n",
       "12                   Year.2007             0.604424           0.604424   \n",
       "13                   Year.1994             0.591667           0.591667   \n",
       "14                  Origin.PSP             0.591646           0.591646   \n",
       "15                    Dest.FLL             0.591537           0.591537   \n",
       "16                   Year.2008             0.587823           0.587823   \n",
       "17                    Dest.CAE             0.585480           0.585480   \n",
       "18                  Origin.ATL             0.567194           0.567194   \n",
       "19                  Origin.PIT             0.558928           0.558928   \n",
       "20                    Distance             0.555028           0.555028   \n",
       "21                   Year.1998             0.552688           0.552688   \n",
       "22                    Dest.CHO             0.549932           0.549932   \n",
       "23                   FlightNum             0.549580           0.549580   \n",
       "24                  Origin.IAH             0.549099           0.549099   \n",
       "25                    Dest.PHX             0.543288           0.543288   \n",
       "26            UniqueCarrier.HP             0.537485           0.537485   \n",
       "27                  Origin.PHL             0.533020           0.533020   \n",
       "28                    Dest.HTS             0.524996           0.524996   \n",
       "29            UniqueCarrier.UA             0.524380           0.524380   \n",
       "..                         ...                  ...                ...   \n",
       "285                   Dest.GSP             0.256756           0.256756   \n",
       "286                DayOfWeek.4             0.256518           0.256518   \n",
       "287                 Origin.CRP             0.255690           0.255690   \n",
       "288                   Dest.GRR             0.254057           0.254057   \n",
       "289                   Dest.HPN             0.252856           0.252856   \n",
       "290                 Origin.LAN             0.251714           0.251714   \n",
       "291                 Origin.TPA             0.249546           0.249546   \n",
       "292                 Origin.CLT             0.248593           0.248593   \n",
       "293                 Origin.CVG             0.248328           0.248328   \n",
       "294                 Origin.PVD             0.243274           0.243274   \n",
       "295                   Dest.RSW             0.238737           0.238737   \n",
       "296                   Dest.MAF             0.237673           0.237673   \n",
       "297                   Dest.CAK             0.231845           0.231845   \n",
       "298                 Origin.BDL             0.231663           0.231663   \n",
       "299                 Origin.SNA             0.228553           0.228553   \n",
       "300                 Origin.ONT             0.225977           0.225977   \n",
       "301                   Dest.SAN             0.224082           0.224082   \n",
       "302                 Origin.STX             0.223197           0.223197   \n",
       "303                   Dest.ANC             0.215848           0.215848   \n",
       "304                 Origin.TYS             0.214950           0.214950   \n",
       "305                 Origin.ELP             0.214924           0.214924   \n",
       "306                 Origin.SRQ             0.203653           0.203653   \n",
       "307                   Dest.MKE             0.194508           0.194508   \n",
       "308                 Origin.EYW             0.179053           0.179053   \n",
       "309           Dest.missing(NA)             0.000000           0.000000   \n",
       "310         Origin.missing(NA)             0.000000           0.000000   \n",
       "311           Year.missing(NA)             0.000000           0.000000   \n",
       "312  UniqueCarrier.missing(NA)             0.000000           0.000000   \n",
       "313      DayOfWeek.missing(NA)             0.000000           0.000000   \n",
       "314          Month.missing(NA)             0.000000           0.000000   \n",
       "\n",
       "     percentage  \n",
       "0      0.008451  \n",
       "1      0.007320  \n",
       "2      0.006543  \n",
       "3      0.006272  \n",
       "4      0.006226  \n",
       "5      0.006009  \n",
       "6      0.005731  \n",
       "7      0.005629  \n",
       "8      0.005624  \n",
       "9      0.005557  \n",
       "10     0.005417  \n",
       "11     0.005385  \n",
       "12     0.005108  \n",
       "13     0.005000  \n",
       "14     0.005000  \n",
       "15     0.004999  \n",
       "16     0.004968  \n",
       "17     0.004948  \n",
       "18     0.004794  \n",
       "19     0.004724  \n",
       "20     0.004691  \n",
       "21     0.004671  \n",
       "22     0.004648  \n",
       "23     0.004645  \n",
       "24     0.004641  \n",
       "25     0.004591  \n",
       "26     0.004542  \n",
       "27     0.004505  \n",
       "28     0.004437  \n",
       "29     0.004432  \n",
       "..          ...  \n",
       "285    0.002170  \n",
       "286    0.002168  \n",
       "287    0.002161  \n",
       "288    0.002147  \n",
       "289    0.002137  \n",
       "290    0.002127  \n",
       "291    0.002109  \n",
       "292    0.002101  \n",
       "293    0.002099  \n",
       "294    0.002056  \n",
       "295    0.002018  \n",
       "296    0.002009  \n",
       "297    0.001959  \n",
       "298    0.001958  \n",
       "299    0.001932  \n",
       "300    0.001910  \n",
       "301    0.001894  \n",
       "302    0.001886  \n",
       "303    0.001824  \n",
       "304    0.001817  \n",
       "305    0.001816  \n",
       "306    0.001721  \n",
       "307    0.001644  \n",
       "308    0.001513  \n",
       "309    0.000000  \n",
       "310    0.000000  \n",
       "311    0.000000  \n",
       "312    0.000000  \n",
       "313    0.000000  \n",
       "314    0.000000  \n",
       "\n",
       "[315 rows x 4 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Deep Learning Variable Importance\n",
    "data_dl.varimp(use_pandas=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.axis.XTick at 0x10edeb630>,\n",
       "  <matplotlib.axis.XTick at 0x10dd096a0>,\n",
       "  <matplotlib.axis.XTick at 0x10ee016a0>,\n",
       "  <matplotlib.axis.XTick at 0x10f0995f8>,\n",
       "  <matplotlib.axis.XTick at 0x10f09b048>,\n",
       "  <matplotlib.axis.XTick at 0x10f09ba58>,\n",
       "  <matplotlib.axis.XTick at 0x10f0a04a8>,\n",
       "  <matplotlib.axis.XTick at 0x10f0a0eb8>,\n",
       "  <matplotlib.axis.XTick at 0x10f0a3908>],\n",
       " <a list of 9 Text xticklabel objects>)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABH4AAAGCCAYAAACW3QTcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xm4LFdZL+Dfl4RBZZ4CJCRRBhlUULkBRCAQgYBKkItA\nRCLClShGcUZxIIiI4EVRETSCXEEvQcCLoAKRYQuIQBhVSEgYEjIyhHkOyXf/qNqk06f32b1P9pmq\n3/d56tm7q1ZXre6vqrrr67VWVXcHAAAAgOk5YG9XAAAAAIDdQ+IHAAAAYKIkfgAAAAAmSuIHAAAA\nYKIkfgAAAAAmSuIHAAAAYKIkfgCADVXV2VX14b1dDwAAdo3EDwDbqqoum5u+UlUfr6p3VtVfVdUx\nVTWJz5+qWhtf4933dl12ox6nfUZV/cSC/Wx2unQP1+focbtP2JPb3S5V9eix/ifv7brsblX1e+Nr\n/b69XRcA2FMO2tsVAGCSOslJSSrJgUmuk+R2SX48yaOTvKOqHt7dZ+21Gm6PfS4pshvca29XYCfe\nk+TlC+ZPPSbsulU4ZgHgCiR+ANgtuvvJ8/Oq6oZJ/izJQ5L8a1Xdsbs/uccrx9K6+yN7uw478Z7u\n/t29XYkMCU72D2IFwMqZRFN7APYP3f2JJMclWUtysyQ7dI2pqutW1VOr6v1V9aWq+kxVvbaq7r3R\neqvquKp6Q1V9uqq+PD73N6vqqgvKXlZVr6+qm1TVC6vqY+N23lFVx23jy11Uz6VfW1Vdq6p+tape\nV1XnVtVXxy5z/1hVd95g/euv7eCqem5VnVdVX6+q48fl/2csc1hVnVBV/zm+XxdV1V9W1bUWrHOH\nMX5mulodX1X3HN/7z1XVZ6vqn6rq1hvU75ZV9bKq+lRVfaGq/r2q7j+7vl17ZzdXVbepqhfMvJcX\nVtXfVtUtFpS9VVU9rapOG9/zr1TVR6rqL6rqpnNlX5jk1AytSNa7EV1WVZeudyfaWfeiqrr5om5W\nY90uq6pDq+pxY6y+VFWnzpW7X1W9qqo+Odbzg2Pdr7kN79k36l1VDx+PkS9W1flV9YdVdZWx3H2q\n6t/GfeBTVfU3VXWdBes7r6rOrKprV9Wzx/V8uar+u6oeu5N6PKyq3jTuX1+qqvdW1a+tb3+DbVyr\nqp457r9fq+F8cG4uP+e8eSZWX5t5/tKxH8t/o5tfVX33GIvPjPv3G6rqyA1e04FV9djxGPjM+L6e\nWVUnV9W3Lih7YlW9dXyPv1hD19mf2WDdD6zhPHDhWP/zxrqcsNF7DMC0afEDwB7V3V1Vv5fkqAxJ\noF9aX1ZVhyX5tySHJXlTklcl+ZYkP5Tk1VX1mO5+3uz6quqvkzwyyblJXprkM0nunOTJSe5VVffu\n7svmqnHdJG9J8ukkf52hK9pDkvxdVd20u5+xna95F1/bbZL83vicfxrreliSByS5X1X9UHdfIQkw\nul6Styb5fJKXJbksycfGZevdXP4wyX2SvDLJa5LcM8lPJbl5kh+YW99G3WI6yQ8nOTbJvyR5TpLb\nJvnBJHesqtt296dmXv+3J/mPJNceX89/Jfm2JP8wvhe7rftNVf1gkpdk+MHrlUk+lCHx+KAkP1hV\nd+/u/5p5yo8m+V9J3pDkzUkuSfKdGd6jH6yhpdr6e/qyJJcmOT7J65O8cWY9Hx3/7kr3ovXnPDvJ\nXTO8x/+UZDZJ8btJfivJJ5O8Isknktw+ya8muW9VfV93f2mL211Uh19Kct8M3erWMuw7v5zk+lX1\n6iQvyPC+/sVY10dkOKaOXbC+q2V4n74lyd+Nj380ybOq6pbd/YuzT6iqpyf5lSQfT/LCJF/MsI/9\nQZJ7V9Ux3T07plMnufpYz2tm2Lc+n+QjSZ6R5EeSfH+G4349PrPP30rsZ90pQyzelOTkJIcneXCS\n11XVHbr7QzOv6apjve6Z5OwkfzvW8YixfmtjfTMmt/4lydFJTh/fg69m6IL552N9Hj2z7scmeVaS\nC5L8Y4Z940YZ9ovjk/zlgroDMHXdbTKZTCbTtk0ZEg2XblLmqhkuYC9NcvjM/LUkX0/yo3Plr5Xk\n3Rku+m44M/+R4/ZekuSqc8/5nXH9P7eofkleNDf/8CQXJ/lKkiOWfK1vGNd19yXKbvW1XTPJ9Ras\n56ZJzk/yvo3e+yTPT3LAguXPH8ucneSQmfkHZEgwXZrkjnPP+UiSD8/N+4lxPV9LctTcst8f1/Mr\nc/NfN85/zNz8+87U+/gl3/f17b8ryRMXTN81U/Z6GZKBFyW55dx6viPJF5K8bcF7fJUF273vWM8/\nmZt/9FifJ2xQ3yePz/u+BctuPj735Ln5Lxznn5Pk0AXPu/e4fC3JNeaWPWpc9rQl389Hb1CHJ4/z\nL05yi5n5V8uQhPj6uOwuG8T6tnPzzx3nvz7JQXMx+vC47M4z879/3P6Hklx/bn/95w32s/Vt/HOS\nq28lFlcy9pcmOW5u2WPHZc+cm//0cf5LZ9+HcdlVMnPcZ0j+XpYhaVUz8yvD8XxpkvvNzH9PhnPJ\ndRe8hh3OJyaTyWRajUlXLwD2uO7+WoYLxiS5YZJU1XcluXuSl3X3S+bKfy7DBf3Vk/zPmUWPy/CL\n/KPHdc76vSSfSvLwBVW4NMmvz23jnCR/muHC6xFbf1Ub25XX1t2f75kWMzPzL8hwwXjrqjp0wea+\nluRXe8dWTt9YRZIndff5M+u8LMNFZCVZ2DVlAy/q7rW5eSfPr2es5z2TfLC7r9Clqbtfk+S1W9jm\nrNtnSPDNT3eYKfOTGZJov91zg4l3939naPlxx5rp8tXdF3T3JfMbG+t6RoYkwJ7QSZ7a3ectWPbz\n4/Kf6u4vXOFJ3X+d5L+zeN/flTr8cXd/cGb9X03y9xkSMP/Q3f8x95y/G//efoN1/np3f31mfZ9K\n8pQM+81PzpR79Lj93+3ui2fKX5ahxVEytM5Z5Je7+ys7e2GLXInYr3X3i+bmPTdD0mb2WDgoyQkZ\nEo6PnX0fxu1csn7c13D3w8cmOS9DgqtnynWGllCVHeN8Sa7Yimn9OTucTwBYDbp6AbC3rA+yun4x\nc5fx77Wr6okLyt9ofM5tkqSqvinJd2Xo3vKLVTuM2VoZukTcZsG6PjomeuatZUjCfPdyL2FpW3pt\n66rqrhmSW3cey8yOWdRJDslwUTjr7N58wOx3Lph37vj3ups8d1fWs56ImU8QrHtzhpYTW/U33f2o\nTcqsj4f0PRu89+sJn9sk+UZyo4bxho7PsI9dN8Pd6dZ9cRfquqtO22D+nTPs3z+2wb5/UJKbVNU1\nu/vzV7IOi+J8wfj3XQuWnT/WYVFi8qvd/fYF89fGv7PH3vr/b5gv3N1nVNWFSW5ZVd/cV+zS9sXu\nPmPBNpayi7Hf4T3q7q9V1SdyxWPhthkSkW/u7o9vUpXbZOgyd1GS39kgzl/JFc8bf5ehG9z7q+rF\nGVry/fts4gyA1SPxA8AeV1VXy9C9IxkSN0ly/fHvvcdpkc4wNkgyXExVhhZDv7OTzS0aW2XRGB3J\ncIGVDOPQbKetvrZU1Y9k6ML25ST/mqG7yxcztCC4Z4YWRFdbsJ6LFsyb95kF89ZbHhy4YNlG9d1h\nPd196XiBOrue9fdzo/d9o/nb4foZ9pPHbFLuGuv/VNWfJfnZDAmMV41/11uPPDrJTba/mhvaKJ7r\nx89m+/41Mowfc2V8dsG8r4/r32hZMrSem/eJBfOSxcfe+v8XbvCcCzPE4tpJZhM/u7w/XYnYLzqm\nkuG9mD0W1ge9Pn9B2Xnr541vz87j/I3zRnf/YVV9LMnPZEga/0KSVNUbMrQEfPcS2wVgYiR+ANgb\n7pbhM+ii7l4fYHX9AvJx3f2sJdaxXv7d3X3HLW7/4A3m33hu3dtlq68tGcYi+WqS7+3uM2cXjHcX\nuvsGz9ttgyRfCZ8b/270vm80fzt8NsN7ctvu/sBmhavqxhm617w7yV3nuwvVrt15bL3b3aLvXTvc\n/WrORvH8XIbWMzfeYPm+6oYbzF907H12Ztm52dFN5sqt26VjYDfFft56guiQJcquv66XdPfDlt1A\nd78gyQtquEvfXTMMYv6oDIPI37q7P72VCgOw/zPGDwB7VA3NQX4zw8XZ380seuv4927LrKe7v5jk\nfUluVwtuHb2Jw8a7bM275/h3UfeVK2NLr2108yTvX5D0qS2uZ1/wnvHvXTZYvjtfz1sztPjZKFE2\n7+Zj+VMXXPgfnuHOS/PWx1PZqLXU+oX2zRYs+x9L1mveW5PcsKpuuYvP31uuVlV3WjB//dibbZGy\n/v9R84XHu8TdJMlZvbU7l+0sVrsS+616f4YWWHeoqhttUvZ9Y9m7jOP9bEl3f667X9XdP5VhsPAb\nZBgwG4AVI/EDwB4zXui8OMk9Mtyt6Knry7r7nRluhfygqvrJDZ7/HVU122LgjzJ0d3p+Ve3QPauq\nrlNVi8brOTDJ02pm0Iyq+tYMA+ZekismpK60XXxtZ2cYv2S+RceTsnjcon1Wd5+bYQyXW1TVT88u\nq6pjsmvj+yzreRlax/xuVX3v/MKqOqCq7jEz6+zx791mL7ar6poZBq5e9N1pffyURcnEJHl7hoTC\no+bWeXiGW4DvSguVPxrX+dwF+0iq6luqaisDde9JfzDepjxJUlU3yOXJ4OfPlPvrDK/xt6vqejPl\nD8xwl6tkGEB5Ky4e17koVmePf7cS+y0ZB3N+ToYueM+ZfR/GbV21qq4/U/ZZGRKGfzJ2kc1c+ZtU\n1a1nHh+1wabXW9VtJUkGwETo6gXAbjEzkO4BGbqz3C7Dr81XydBa4ccX3GXmxzLcCvq5VfXzSd6W\noWvEoRkGWr1dhlYjn0iS7n5+VX1Phu4ZH6qq1yT5aIbxT741QyuPvx6Xz/rPJHdK8s6qOjXDeEE/\nmmGskF/t7o9s5aUm+Y2NEjoZbv/8nq2+tiR/nOEC8T1V9bIMCam7Zkj6vCLJD2+hjrvDDiPNbuJn\nk/x7kmdX1f0zxODbMnRDeXmSY3N5l6ht092frKofzXAntLdX1WsztLroDBf/35fhIvxaY/nzq+ql\nGe6w9q6x/LWT3CdD64v/yo6Jt/dnGG/m4VXVGfbBTvJ/uvv87n5LVb0lQ8uVt4/jrRw8vuZ/SfLQ\nXXhd/1pVv5mhS+BZVfWqJB8ZX8sRGZKrr0/ygK2uewu2ug8kw2Dk10zy31X1ilx+N7uDMxwr663j\n0t1vqqo/SvKLSd43xuVLSX4wQwzWMhwnW/H6DLF5elXdIcMxeFl3P3UXY78rnpihpdcDk5xZVf+U\n4S5fh43belyS/ztT9jsznMOOrarXZxhY++Akt8yw//5ahjuOJckrq+rTGc6xZ2dIct8tyfdmOOfs\nMFA2ANMn8QPAdltvvbA+GOnXMlw0nZPkb5K8tLv/deEThwuv703ycxkuvn4sw4XLRRkurv8kw8XX\n7HN+brzo/ekMLUeuk+E27h9N8rQsbr3z6ST3S/L0JI/McNH/viT/u7tfvAuv9z47Wf7/krxnq6+t\nu0+uqq9kGJz1+AyDPL9xrO+Dszjx09m89cjOlm/0/GXnbbie7j69qu6c5PeT3CtD157/TPIjGe50\ndGwuHwtoGcu81vVt/2tV3T7D7a/vk+FC+KsZkjWvyZAUmvUTSc7KkAx8bJKPZ0hO/U6Sf1rw2i6t\nqgdmuJvSQzIkNpLhInt9EN8fyrC/PSBDEuzMDBf4bxqfs9X3ON391Kp6U4aWat83rvuzGZIrz87l\nyYNlLP1+Llm/jdb31Qzxf2qS4zIMYPyhJE/u7ufssJLuX6mqd2aIw/EZEscfTPIbGW4zv8Nty3dW\nr+5+X1U9Mskvjeu8eoYBmNdbH24p9pu81oX16e6vVtV9MgzA/IgMx3Qy7CsvSfKWmbJfT/KAqnrE\nWLcfypDc+0SSDyd5QpJTZlb/qxn28e9Jcv8MA1OfPc7/i+7e9uQqAPu+6t78M35shv3MDL/aPq+7\nnza3/DoZflG9eYYvpo/q7vdvf3UB4MqpqsuSrHX3vfZ2XRhU1d8leViSW3f3WXu7PuweVXVuki93\n9632dl0AYJVs2ld57OP8rCT3zdAM/bjZvsSjJ2S4q8rtM/wa8afbXVEAYP9Vgx3u3lVVR2do8fI+\nSR8AgO23TFevIzPcMeGcJKmqUzI0xz5jpsxtMzaR7e4PVNURVXXD7v7EDmsDAFbRVZOcO45vc0aG\n7jW3S3LvDN1/fnYv1g0AYLKWuTvBIUnOnXl83jhv1nszDM6Y8Q4Sh2UYrBIA9jW7MpYJV94lGQar\nvmmGsVpOzDBo7YuT3KW737QX68ae49gDgD1suwZ3/oMMt5l8V4aBKd+dZNFgewCwV3X3gXu7Dqto\nHFT2cXu7Huw93X2zvV0HAFhFyyR+zs/Qgmfdobn8DhVJku7+fJJHrT+uqo9kuNPAFYy3OAUAAABg\nG3V3LZq/TFev05LcoqoOr6qrZrjrxitmC1TVtavqKuP/P5Xk37r7CxtUZL+envjEJ+71OpjE2yTe\nJvE2ibdJvE3ivYqTeK/WJN7LTzuzaYuf7r60qk5Mcmouv5376VV1wrC4T05ymyR/M94i931JHr3Z\negEAAADYvZYa46e7X53k2+fm/eXM/2+dXw4AAADA3rVMVy9mHHXUUXu7CuxB4r1axHu1iPdqEe/V\nIt6rRbxXi3ivFvHeHrVZX7Bt3VhV78ntAQAAAExdVaWvxODOAAAAAOyHJH4AAAAAJkriBwAAAGCi\nJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiDtrTG6xaeFt55hx8+OG56Oyz93Y1AAAAgP1Y\ndfee21hVZw9ub79WlT0ZGwAAAGD/VEMOYWFLG129AAAAACZK4gcAAABgoiR+AAAAACZK4gcAAABg\noiR+AAAAACZK4gcAAABgoiR+AAAAACZK4gcAAABgoiR+AAAAACZK4gcAAABgoiR+AAAAACZK4gcA\nAABgoiR+AAAAACZK4gcAAABgoiR+AAAAACZK4gcAAABgoiR+AAAAACZqqcRPVR1TVWdU1ZlV9fgF\ny69fVa+qqvdU1X9V1SO3vaYAAAAAbEl1984LVB2Q5MwkRye5IMlpSR7W3WfMlHlikqt3929U1Q2S\nfCDJwd399bl1dTbZHqOqbBYbAAAAgBpyCLVo2TItfo5MclZ3n9PdlyQ5Jcmxc2UuSnLN8f9rJrl4\nPukDAAAAwJ510BJlDkly7szj8zIkg2b9VZLXVdUFSa6R5KHbUz0AAAAAdtV2De78G0ne2903TfLd\nSf68qq6xTesGAAAAYBcs0+Ln/CSHzTw+dJw3665JnpIk3f2hqvpIklsneccOazvppMv/P+qoYQIA\nAABgKWtra1lbW1uq7DKDOx+YYbDmo5NcmOTtSY7r7tNnyjwjyee6+0lVdXCGhM/tu/tTc+syuPOy\nDO4MAAAALGFngztv2uKnuy+tqhOTnJqha9jzuvv0qjphWNwnJ3lqkudX1XuTVJJfm0/6AAAAALBn\nbdriZ1s3psXP8rT4AQAAAJZwZW/nDgAAAMB+SOIHAAAAYKIkfgAAAAAmSuIHAAAAYKIkfgAAAAAm\nSuIHAAAAYKIO2tsVYLpufMQR+dg55+ztauwXDj788Fx09tl7uxoAAABMTHX3nttYVWcPbm+/VpU9\nGZvdoaoi3kuaQLwBAADYO2q4pqxFy3T1AgAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAA\nAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiTpob1cAmIYbH3FEPnbOOXu7\nGvuFgw8/PBedffbergYAALACqrv33MaqOntwe/u1quzJ2OwOVRXxXpJ4r5YJxBsAANh31HCNUYuW\nafEDwJZp4bU8LbwAANibtPjZV02gRYAWIFsg3qtFvFfLBOINAMC+bWctfgzuDAAAADBREj8AAAAA\nEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBRSyV+quqYqjqjqs6sqscvWP4r\nVfXuqnpXVf1XVX29qq6z/dUFAPa0Gx9xRKrKtMR04yOO2NvhAgC4gurunReoOiDJmUmOTnJBktOS\nPKy7z9ig/A8l+YXu/oEFyzqbbI9RVTaLzb6uqiLeSxLv1SLeq0W8V8sE4g0A7H9q+A5Si5Yt0+Ln\nyCRndfc53X1JklOSHLuT8scledHWqwkAAADAdlom8XNIknNnHp83zttBVX1TkmOSvOzKVw0AAACA\nK+OgbV7fDyd5c3d/ZsMSJ510+f9HHTVMAAAAACxlbW0ta2trS5VdZoyfOyc5qbuPGR//epLu7qct\nKPsPSf6+u0/ZYF3G+FnWBMYIMCbEFoj3ahHv1SLeq2UC8QYA9j9Xdoyf05LcoqoOr6qrJnlYklcs\n2Mi1k9wjyT9emcoCAAAAsD027erV3ZdW1YlJTs2QKHped59eVScMi/vksegDk7ymu7+8+6oLAAAA\nwLI27eq1rRvT1Wt5E2gqrmvAFoj3ahHv1SLeq2UC8QYA9j9XtqsXAAAAAPshiR8AAACAiZL4AQAA\nAJgoiR8AAACAidr0rl4AAKyOGx9xRD52zjl7uxr7hYMPPzwXnX323q4GAOyUu3rtqyZwVxB3gdkC\n8V4t4r1axHu1iPdqmUC8AZgGd/UCAAAAWEESPwAAAAATZYwfAABYUcZ0Wt4UxnQS7+VNId6wzhg/\n+6oJ9Bk3RsAWiPdqEe/VIt6rRbxXi3ivFvFeLROIN6vFGD8AAAAAK0hXLwAAAJgYXfuWN/Wufbp6\n7asm0LRQU9ItEO/VIt6rRbxXi3ivFvFeLeK9WsR7tUwk3rp6AQAAAKwYiR8AAACAiZL4AQAAAJgo\niR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAA\nAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJiopRI/VXVMVZ1RVWdW1eM3KHNUVb27qv67\nqt6wvdUEAAAAYKsO2qxAVR2Q5FlJjk5yQZLTquofu/uMmTLXTvLnSe7T3edX1Q12V4UBAAAAWM4y\nLX6OTHJWd5/T3ZckOSXJsXNlfizJy7r7/CTp7k9ubzUBAAAA2KplEj+HJDl35vF547xZt0pyvap6\nQ1WdVlWP2K4KAgAAALBrNu3qtYX1fE+SeyX5liT/UVX/0d0f3Kb1AwAAALBFyyR+zk9y2MzjQ8d5\ns85L8snu/kqSr1TVG5PcPsmOiZ+TTrr8/6OOGiYAAAAAlrK2tpa1tbWlylZ377xA1YFJPpBhcOcL\nk7w9yXHdffpMmVsn+bMkxyS5WpK3JXlod79/bl2dTbbHqCqbxWZfV1UR7yWJ92oR79Ui3qtFvFeL\neK8W8V4t4r1aJhLv7q5FyzZt8dPdl1bViUlOzTAm0PO6+/SqOmFY3Cd39xlV9Zok/5nk0iQnzyd9\nAAAAANizNm3xs60b0+JneRPJOIr3ksR7tYj3ahHv1SLeq0W8V4t4rxbxXi0TifdGLX6WuasXAAAA\nAPshiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4\nAQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACA\niZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8A\nAACAiZL4AQAAAJgoiR8AAACAiZL4AQAAAJgoiR8AAACAiVoq8VNVx1TVGVV1ZlU9fsHye1TVZ6rq\nXeP0W9tfVQAAAAC24qDNClTVAUmeleToJBckOa2q/rG7z5gr+sbufsBuqCMAAAAAu2CZFj9HJjmr\nu8/p7kuSnJLk2AXlaltrBgAAAMCVskzi55Ak5848Pm+cN+8uVfWeqvrnqrrtttQOAAAAgF22aVev\nJb0zyWHd/aWqul+Slye51TatGwAAAIBdsEzi5/wkh808PnSc9w3d/YWZ/19VVc+uqut196d2WNtJ\nJ13+/1FHDRMAAAAAS1lbW8va2tpSZau7d16g6sAkH8gwuPOFSd6e5LjuPn2mzMHd/bHx/yOT/H13\nH7FgXZ1NtseoKpvFZl9XVRHvJYn3ahHv1SLeq0W8V4t4rxbxXi3ivVomEu/uXjj28qYtfrr70qo6\nMcmpGcYEel53n15VJwyL++QkD66qn0lySZIvJ3no9lUfAAAAgF2xaYufbd2YFj/Lm0jGUbyXJN6r\nRbxXi3ivFvFeLeK9WsR7tYj3aplIvDdq8bPMXb0AAAAA2A9J/AAAAABMlMQPAAAAwERJ/AAAAABM\nlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAA\nAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ\n/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAAwERJ/AAAAABMlMQPAAAA\nwERJ/AAAAABM1FKJn6o6pqrOqKozq+rxOyn3P6rqkqp60PZVEQAAAIBdsWnip6oOSPKsJPdNcrsk\nx1XVrTco9wdJXrPdlQQAAABg65Zp8XNkkrO6+5zuviTJKUmOXVDu55K8NMnHt7F+AAAAAOyiZRI/\nhyQ5d+bxeeO8b6iqmyZ5YHc/J0ltX/UAAAAA2FXbNbjzM5PMjv0j+QMAAACwlx20RJnzkxw28/jQ\ncd6sOyY5paoqyQ2S3K+qLunuV+ywtpNOuvz/o44aJgAAAACWsra2lrW1taXKVnfvvEDVgUk+kOTo\nJBcmeXuS47r79A3KPz/JK7v7HxYs62yyPUZV2Sw2+7qqingvSbxXi3ivFvFeLeK9WsR7tYj3ahHv\n1TKReHf3wt5Xm7b46e5Lq+rEJKdm6Br2vO4+vapOGBb3yfNPudI1BgAAAOBK27TFz7ZuTIuf5U0k\n4yjeSxKpNED7AAAbC0lEQVTv1SLeq0W8V4t4rxbxXi3ivVrEe7VMJN4btfjZrsGdAQAAANjHSPwA\nAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBE\nSfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAA\nAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATJTE\nDwAAAMBESfwAAAAATJTEDwAAAMBESfwAAAAATNRSiZ+qOqaqzqiqM6vq8QuWP6Cq3ltV766qd1TV\nvba/qgAAAABsRXX3zgtUHZDkzCRHJ7kgyWlJHtbdZ8yU+ebu/tL4/3cm+X/dfYsF6+pssj1GVdks\nNvu6qop4L0m8V4t4rxbxXi3ivVrEe7WI92oR79UykXh3dy1atkyLnyOTnNXd53T3JUlOSXLsbIH1\npM/oGkk+uauVBQAAAGB7LJP4OSTJuTOPzxvnXUFVPbCqTk/yL0l+fnuqBwAAAMCu2rbBnbv75d19\nmyQ/nOSF27VeAAAAAHbNQUuUOT/JYTOPDx3nLdTdb66qg6rq+t198Q4FTjrp8v+POmqYAAAAAFjK\n2tpa1tbWliq7zODOByb5QIbBnS9M8vYkx3X36TNlbt7dHxr//54kL+numy9Yl8GdlzWRwaXEe0ni\nvVrEe7WI92oR79Ui3qtFvFeLeK+WicR7o8GdN23x092XVtWJSU7N0DXsed19elWdMCzuk5P8z6o6\nPsnXknwxyUO3r/oAAAAA7IpNW/xs68a0+FneRDKO4r0k8V4t4r1axHu1iPdqEe/VIt6rRbxXy0Ti\nfWVu5w4AAADAfkjiBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkri\nBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAA\nJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4A\nAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJkriBwAAAGCiJH4AAAAAJmqpxE9VHVNV\nZ1TVmVX1+AXLf6yq3jtOb66q79z+qgIAAACwFZsmfqrqgCTPSnLfJLdLclxV3Xqu2IeT3L27b5/k\n95L81XZXFAAAAICtWabFz5FJzuruc7r7kiSnJDl2tkB3v7W7Pzs+fGuSQ7a3mgAAAABs1TKJn0OS\nnDvz+LzsPLHzv5K86spUCgAAAIAr76DtXFlV3TPJTyb5/g0LnXTS5f8fddQwAQAAALCUtbW1rK2t\nLVW2unvnBarunOSk7j5mfPzrSbq7nzZX7ruSvCzJMd39oQ3W1dlke4yqslls9nVVFfFeknivFvFe\nLeK9WsR7tYj3ahHv1SLeq2Ui8e7uWrRsma5epyW5RVUdXlVXTfKwJK+Y28BhGZI+j9go6QMAAADA\nnrVpV6/uvrSqTkxyaoZE0fO6+/SqOmFY3Ccn+e0k10vy7KqqJJd095G7s+IAAAAA7NymXb22dWO6\nei1vIk3NxHtJ4r1axHu1iPdqEe/VIt6rRbxXi3ivlonE+8p09QIAAABgPyTxAwAAADBREj8AAAAA\nEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8A\nAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBR\nEj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAAADBREj8AAAAAEyXxAwAA\nADBREj8AAAAAEyXxAwAAADBRSyV+quqYqjqjqs6sqscvWP7tVfWWqvpKVf3S9lcTAAAAgK06aLMC\nVXVAkmclOTrJBUlOq6p/7O4zZopdnOTnkjxwt9QSAAAAgC1bpsXPkUnO6u5zuvuSJKckOXa2QHd/\nsrvfmeTru6GOAAAAAOyCZRI/hyQ5d+bxeeM8AAAAAPZhBncGAAAAmKhNx/hJcn6Sw2YeHzrO2zUn\nnXT5/0cdNUwAAAAALGVtbS1ra2tLla3u3nmBqgOTfCDD4M4XJnl7kuO6+/QFZZ+Y5Avd/YwN1tXZ\nZHuMqrJZbPZ1VRXxXpJ4rxbxXi3ivVrEe7WI92oR79Ui3qtlIvHu7lq0bNMWP919aVWdmOTUDF3D\nntfdp1fVCcPiPrmqDk7yjiTXTHJZVT0uyW27+wvb9zIAAAAA2IpNW/xs68a0+FneRDKO4r0k8V4t\n4r1axHu1iPdqEe/VIt6rRbxXy0TivVGLH4M7AwAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8\nAAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADA\nREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8A\nAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyUxA8AAADAREn8AAAAAEyU\nxA8AAADARC2V+KmqY6rqjKo6s6oev0GZP62qs6rqPVV1h+2tJgAAAABbtWnip6oOSPKsJPdNcrsk\nx1XVrefK3C/Jzbv7lklOSPIXu6Gu+4a1tb1dA/Yk8V4t4r1axHu1iPdqEe/VIt6rRbxXi3hvi2Va\n/ByZ5KzuPqe7L0lySpJj58ocm+QFSdLdb0ty7ao6eFtruq+w460W8V4t4r1axHu1iPdqEe/VIt6r\nRbxXi3hvi2USP4ckOXfm8XnjvJ2VOX9BGQAAAAD2IIM7AwAAAExUdffOC1TdOclJ3X3M+PjXk3R3\nP22mzF8keUN3v3h8fEaSe3T3x+bWtfONAQAAALBl3V2L5h+0xHNPS3KLqjo8yYVJHpbkuLkyr0jy\ns0lePCaKPjOf9NlZJQAAAADYfpsmfrr70qo6McmpGbqGPa+7T6+qE4bFfXJ3/0tV3b+qPpjki0l+\ncvdWGwAAAIDNbNrVCwAAAID9k8GdgU1V1c2q6nNVdePdUR6AvcP5nT2tqu5RVZfs7XpMnWMbmLVU\n4qeqnltVr5+bV1X1xqp6zu6p2g51eEBVvaGqPjlOrx/HE5ot8+1jmS9U1TljF7XZ5deoqhdU1Weq\n6uKqek5VXWVm+S9V1Zkzy99QVXfbE69vX1ZVa1X1lar6bFV9uqo+WFUvrKrv3ab1f6SqfmyTMk+s\nqkvGD6TPVdXnx78/XVXXqaqvV9VdZ8pfe5z3orn1/E1VvXT8/6S5dX64qp6yHa9pX1ZVd6mqV437\n+eer6rSqOn5nz+nuc7v7Wt190TLb2Gr5ndT1+Kp6x1jPT4/1vstcmfX983Pja3pvVT18gzK7ZR/e\nn6zQ+fy+VfW6qvpUVV1WVdfbE69tX7NC8X58Vb17PMYvqKq/raqb7onXty/ZX87vVXWvqvq38fj8\ndFWdXlVPnln+/Kr62szn8+eq6tUzy69WVU+uqrOq6ktVdeH4nEN3tU6rZvxcvKyqHjw3/8hx/oe3\naTs/UVVnLVi0pS4HVfXQ8bz12ar62nbUbX+yvxzbM/X916q6tKoOm5u/MOk3P7+qbjB+fp03Hv/n\nV9U/V9XBV7ZuU7G7v9vWctdn31RVf1RVZ49xuqiqXltVtxuXHz6eT9av29b/3n1mHVvet9k1y7b4\neVySQ6vql2fmPT7JDZL84nZWaPxCeuCCRddO8owk35rkRkn+MclrqupG4/OukuSfk7w9yfWSPDjJ\nk6rqh2fW8ZwkNxvXcdsk35vkD2aWvzzJnbr7OklunGFco3/aoD6rpJP8bndfu7uvm+SeSc5O8h9V\ndewerMcbxg+ka3X3Nce/f9Hdn0nyriT3mil7VJLTx7+z7pXkX8f/e3adSY5P8gtV9VO79VXsRVV1\nnySvT/LvGY6DGyZ5epJnVtUTN3jOVRbN392q6klJ/jjDMXrDJN+W5C1JXl9VPzBTdH3/vFaGY/+v\nk/xNVX37gjJ7ex/eF6zK+fwLGfaFR2aLFxgTsyrxPjDJCRle1/qx//+255XtH/aX83tVHZHklUn+\nMkMdr5/kQUnOmCv6f2Y+8681c3fZA5L8S5IHZrjZyDWS3DnJtZK8rapusidexwR0kvcnmf/O81Pj\n/O1S2Z5z8KeS/HmSX9iGde1X9pdje2bb35bh+/d/Zcf9K9l4f5id/3cZju3bj9/vbp/kRTt57ira\nF77bPjPJdyf5/jFOt8pwnH59rp63mrt+e2Oya/s2V0J3LzUlOTLJ55J8R5I7JPl8kjuMy66f4Qv2\nuUkuSvJ/k9xg5rm/kOED/XNJPpLkyTPLDkxyWZITk7wzw+DQ37NknT6d5Jjx//sl+UySq84sf3qS\nfx7/v2aSrya588zy+4/POWDBuq+W5NeSnLHsezTVKckbkjxhwfznJvno+P83JfnfST6c5JMZvpTd\nfKbswzJ8kfhshrvDPX+c/4oklyb50rh/vHqDOjwxyak7qePvJ1mbefynGS5uzkjyHeO8bx+3dfON\n1pnhwuPP9vZ7vhtjeWaS5y6Y/xNJvpbksCTPT/K349+LM5zADx+P05vOPOcJ4zH/ySR/lOS1SX5n\nXHaF8uN7/dokT0nysfE8cdJO6nl4kkuS/PiCZc9P8oGN9s8k3zxu+39uZR9epSkrdD6fOe6vt7ff\nd/He/fGeWX77Me5X39vv/x6M8/5yfn9Qkgs3eS3PT3LyBst+PMlXknzr3PyDknxgo+eZdngf35Dk\nd5N8PMkR47xrZEiw/HKSD4/zvinJnyT56Fj2H5LcbG49/zvJS8fzxFlJHjAuu3OSL2e4CPz8uPzu\nSe4x7pMPSfLB8Xzw4iTfskS975Hka3v7/dvDsdovju2ZbfxBktcleUSSCzJznt4ofvPzx33lmL39\n3u/LU/aN67P/TPLTO6nj4eN6brrB8p3t25ckOWxvv89TmpYe46e7357kaRm+FL4wQ4bxPePiV2b4\nEL51kiPG//925ukfTXKfHjKBP5LkhKqav/PXT2b4MnCNDBninRqbsX1Lkv8eZ31Xkvd392zzz3dl\n+PKXJLfL8KXgXXPLr5mhJcH6eu9VVZ/OsKM/JsnsL45c0SlJDqmqW2U4ydwqwwXGjZO8LWNrqar6\npiQvSPIz3X3tDO/3c5Okux+QYf94dM/8orcLXpfkzuO2kuToDBnktfH/9XnndveH5p88/lJ9zwz7\nyVt2sQ77tKq6ZZJbZPgVZd7/zdAC8N7j4wdn+AX+Bhm+ACYzv7KMTTB/LskPJjk4w4fFN5ptzpcf\n3S3DLxE3SXJskifUXLetGfcZn3/KgmUvTHKLqrr5/IKqumqSx2a4SHznBuueNbsPr4xVOZ8zWNF4\n/0CSD3b3VzarzxTsZ+f3dyRZ77p3bG29e9b9krytuz9yhQp1fz1D8uB+W1zfKvtKhn3mf42Pj8vw\nvWm2q88zM3y3OzLDRdzFSV5ZVTVT5vgkfzieJ/48Q6vbq3f3W5P8dIYk0hV+6c9wTN87yXdm+P74\n3Ul+fvtf4v5tPzu2U1UHZWhp+8IMScJrjc/bqn9L8odV9VNVdYexpR/L2ZPXZ29M8utV9fM1dBO9\n6rKVXGLfrly+b7MNtnoQPTXDh8Tnu/sPk6Sq7pThV8Sf7+4vjl+yfiPJfdabcXf3P3T3R8f/35Mh\nwEfPrfvp3X1OD3Y64FsN/fb/PsMvj+eNs6+ZIVs56zMZTjjJ8IX0a3NfLD+TYadaL5Pufn0PzeWu\nn8u7eu215pL7uPX3/sYZviw8trs/OX75enKGD4k7jWW+luQ2VXXd7v5yd//73LoqmzuqhvEA1scE\n+FRVfd+47M0ZfqW4Ww2D0t0kw5fL1+fyfe1eGX652GGdGX6Rem2Sp3T3izJNN8zwgX7+/ILxmPtk\nhm4YSfLm7n7peDwuunB6RJK/7O7/7O5Lx/PBBZts/8zu/qvuvqy735bkPUnuuJO6ru9L8y7IsL/c\naGbeb45x/FKSJyX5oe4+e5P6JMM+XBmO91Uz+fM5V7Ay8a6qo5L8TpKf2VldJma/Ob+P+9OdMuyP\nf5jk7BrG+Jm/ODx+7vN+fSyaGy56naMLcsXPBjb33CSPHLtpPibJX60vGJM7xyf5ze6+qLu/nKEV\n4G0yXEiue/EY9yQ5OUP3zltust1O8vjxO+EnMgy3sNF3glW23xzbowdlOKe/tLu/mKHL7WM22cYi\nD83wI8QjM3QDuriGsWSWTiyssD15ffa4DC10j80wlManahhv7Tpz63jfzPn8HeP8rezbbIMtJX66\n+7Ik78vlv9IlQ3+8b07y8fWL8gzNwr+coelhqurhNQzUdPHYmuaEDMGedc4ydaiqm2Vo2vay7n7y\nzKLPZ/igmXWdDM3T1pdfde6Esb5Tfi5zehg35ufH17DyAzxvYP1XukvHv/85sw9cnOHXnJuNXxTu\nn+FXuA+N+8Jxu7C9te6+3jhdd/z7liTp7q9maKlzr3F6U3d3hn3lbmPy7qjsmPhZ6+7rZbiweEKG\nMX6mOnDcJzKcfA+ZXzC+PzcYyyTDrzs7c0h2PGbP3eQ5F849/mKGLweLfCLJDcZfjubdNMMHxSdm\n5j1ljOP1k7wqw0XfMg4d13XxkuUnY5XO56xOvKvq6Ay/Mv9Ed19hUOuJ25/O7+nu93f3Y7r7Vhla\nHrw6yUuq6hYzxV4w93n/0nH+J7LgdY5umit+NrCJ7n5fhnj/doZj+9Uzi2+YYeiDs2fKfzFDl6+b\nzZS7cGb5l8Z/N4z/6NLu/tTM453uMytsvzq2MyR5XtndXxgf/22GHxOOGB9fkuSAuRZjSXKVcVmS\nYT/q7qd1910zfD48IsmjMnxXZ+f22PXZmEB8dncfneFz+f4ZrsOeOVssyW1nzufricOt7Ntsg+1o\nNndOks/OXJCvB/Vbuvsd44H+ggwXYjfsoTXNX2bHDOJlm21o/ELwxiSndPevzS1+b5LbzrXO+Z5x\nfjL0X/z6OG/d92b40rjRnQsOHKfPb1a3FfWwDFnaszIc1Lec2weu0d0vTpLufmN3H5vhwvwpSf62\nqr51XM+msV/SazM071/v5pXxV6RzM3wQXSdDl7AddPfXu/tpGfaTJy8qs7/r7rMy7OuLRug/LkMc\n1ge+3iwm52do8j3rZosK7qL1ejx0wbIfT/Kh7v7g/ILu/myGgQS/sza5E8HoYUnO7+4zd7mm0zLl\n8zk7mlS8axgM+iVJHt7dL9+sTlOyn53fr6C7L86QdDgoQwu0zbw6yZ1mvkMkScYWKw/JMIYFW/NX\nSX4ryfPGH83WfSJD1+kj1mdU1TUy/Ar/0SXXvV3f8VbS/nRsj13w75nkfjXcae/CJH8zLl4f5Pns\nDJ8h8910b5kNPr/H7+j/lOF7/h22q74Ttleuz8aWZm/M8Dk8H6cdWg5tcd9mG2xH4udtSU6vqmdW\n1XWTpKpuVFUPGZdfY/z7ye6+bOya8/BFK9qZqvqODF8a/6q7n7igyOsyfEA9uYbbfB6Z5NFJ/iJJ\nuvvzGfo8PqWqrl/DXR+emOFD7rJxGz8zzk9V3TDDXUTOzNCskVFVHVrDHZeOz9BF4OMZ+mI+Z2zG\nnxpusf7AqvrmcX94UFVda/xC8dkMJ6L1TPRF2bxJ8DJel+FEc/+MiZ/RWpLfTPLfYyJoZ34rQ5Pn\nqY4TcmKSH6+qJ1TVdavq6mPz+T9O8gfdvbNf7mdP2i9M8piqun1VHVTDHYPm76SyTPe9hXropvX0\nJH9SVQ8e63ndqvrNDB8GJ+7kuZ/OMGDhSbVBn/D5fXhX6zlBUzqfV1VdLcOv1ZXk6uNjLjeleB+X\n4bz0kO5+1VbrOBH7xfm9qr6/qn6uqm42HqfXSPLrGbrqvmOTpyfD943/SPLyqvofVXXAmKQ8JUPX\nvyftat1W2IsyjK33p7Mzx+9sL8hwbN6kqr45wx36Tk9y2pLrvijJjarqSrXmGeO8fk7PeK5YlXP6\nfnFsZ2gR+uEM3+lvP07fleEH1UdV1YHdfUGGBM4zamxhX1V3yDAm0fO/UYmqZ1TVHcc4Vw1deO+Z\n4bOEBfbG9VlVnVRV/7+9u2eNKojCOP6cFIKmFQQLv4ONqBAUCzGFhSAIQlIIVtoYFIxoI4KVBlH8\nAIKgIDZiFAtfQEExamMbUqiFsdAkvhBEjsUzSW7WbDabFLJ3/z/YInd3Z2925s69O/fMmb6I6C1/\nb5XzAVbrabk2tZa2jTateeCnXHTtl7RO0tuI+CbPxewrz7+XD/jRcJj4kNwIFxXTWG5EHImIycqm\nM3I48OmImCmP6Yg4UD7nt5ysbLu8IsEdOUv9vUoZx+Toj3E55H1MvtiYs1PSm4iYkRPD9sgZ5ZfN\nWdAlzkXEVKnfZ/JI/Y7KXdWj8hSBpxExJd+pPSjXbY/83U+U565KGsySN0LSBUkD4akE9yUpIoYj\nojFJ6O5S59OV+r9YeX5MvgMcmVl972O57bQcNc7M5/JA0UqnCnWUzHwoR0Ttku+6fJFzegxlZqv/\nef44zcwbcgLHUfnEsFnSS/nO4D+vb1WeJJU6nQ8xzcyz8oXAcNnPCblf2ZOZj1p8zhV5WejByrZW\nbbjr1aw/3ytPWXpX9umjpJ9lUAGqXX1fkqet3W04R2xs+4vpUB3Uv3+Vp16/kM/Z43K+mP5cyPvU\nvODMP5L2yavO3JL0vezfD0nbMrNZ/h8sVq3z2XSOy8ZcW5J0Qj7eXsvtapO8atfc+5dqC9VtT+Tr\nr4nwdJMVpU+IiNGIuF7ZNCD36Q/kaPxfcp++ZSXldbJOOLbDEZuDkkYyc7L6kKf99GohyfPh8vmv\nynnnZtmvkUrRPfKKk5/l88I1OZ/c5Rb7123+9++zWbl+P5Qybst5/E5VXtO0Ta2xbaNNsTiiEwDa\nFxEh/yg7mZlLrcQFAOhA9O9APXFsA92FpfEArEpEHCohuBsknZe0Xr4TBwDoYPTvQD1xbAPdi4Ef\nAKt1XA7V/SSH7fc3CRMHAHQW+negnji2gS7FVC8AAAAAAICaIuIHAAAAAACgphj4AQAAAAAAqCkG\nfgAAAAAAAGqKgR8AAAAAAICaYuAHAAAAAACgphj4AQAAAAAAqKm/61tx9UmaQZkAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ebffac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot Deep Learning Feature Importances\n",
    "all_coefficient_magnitudes = data_dl.varimp(use_pandas=True)\n",
    "importances = all_coefficient_magnitudes[1:10]\n",
    "feature_labels = list(importances['variable'])\n",
    "Index = importances.index\n",
    "\n",
    "plt.figure(figsize=(20,6))\n",
    "h = plt.bar(range(len(feature_labels)), importances['relative_importance'],width=0.6, label=feature_labels, color ='aqua')\n",
    "plt.title(\"Deep Learning Feature Importances\",fontsize = 20)\n",
    "xticks_pos = [0.65*patch.get_width() + patch.get_xy()[0] for patch in h]\n",
    "plt.xticks(xticks_pos, feature_labels,fontsize = 13,   ha='right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"question\"></a>\n",
    "# Can H2O Handle New Categorical Levels in a Test Set? \n",
    "___\n",
    "> *Yes! Unlike most machine learning algorithms, H2O-3's algorithms can successfully make predictions, even if a test set contains categorical levels that were not present in the training set. This is because every algorithm handles new categorical levels specifically. So, the next question becomes:* \n",
    "\n",
    "> __How does each algorithm handle unseen categorical levels in a test set?__ \n",
    "\n",
    "### Skip to the algorithm you're using to see how it predicts on a categorical level not seen during training:<a name=\"answer\"></a>\n",
    "\n",
    "* [Generalized Linear Model (GLM)](#GLM)\n",
    "* [Gradient Boosting Model (GBM) & Distributed Random Forest (DRF)](#GBM & DRF)\n",
    "* [Deep Learning](#DL)\n",
    "* [K-Means](#K)\n",
    "* [Naive Bayes](#NB)\n",
    "* [Principal Component Analysis (PCA)](#PCA)\n",
    "\n",
    "\n",
    "___\n",
    "\n",
    "<a name=\"GLM\"></a>\n",
    " ** GLM **:\n",
    "\n",
    "  GLM will predict 'Double.NAN' for each row with a new categorical level, indicating a prediction wasn't made.\n",
    "  \n",
    "  *After running the cells to load, clean, and split the data you can play with a GLM [here](#GLM Model).*\n",
    "\n",
    "<a name=\"GBM & DRF\"></a>\n",
    " ** DRF & GBM ** :\n",
    "  \n",
    "  Unseen factors can go either left or right for small counts of factor levels. Otherwise, for large counts, they go\n",
    "  left. \n",
    "\n",
    "  *After running the cells to load, clean, and split the data you can play with a GBM [here](#GBM Model) or a DRF [here](#DRF Model).*\n",
    "\n",
    "\n",
    "<a name=\"DL\"></a>\n",
    " ** Deep Learning **:\n",
    "\n",
    "  For an unseen categorical level in the test set, Deep Learning makes an extra input neuron that remains untrained and contributes some random amount to the subsequent layer.\n",
    "\n",
    "  *After running the cells to load, clean, and split the data you can play with a Deep Learning model [here](#DL Model).*\n",
    "\n",
    "<a name=\"K\"></a>\n",
    " ** K-Means **: \n",
    "\n",
    "  An unseen categorical level in a row does not contribute to that row's prediction. This is because the unseen  categorical level does not contribute to the distance comparison between clusters, and therefore does not factor in  predicting the cluster to which that row belongs.\n",
    "\n",
    "<a name=\"NB\"></a>\n",
    " ** Naive Bayes **:\n",
    " \n",
    "  If the Laplace smoothing parameter is disabled ('laplace = 0'), then Naive Bayes will predict a probability of 0 for any row in the test set that contains a previously unseen categorical level. However, if the Laplace smoothing parameter is used (e.g. 'laplace = 1'), then the model can make predictions for rows that include previously unseen categorical level.\n",
    "  \n",
    "  Laplace smoothing adjusts the maximum likelihood estimates by adding 1 to the numerator and k to the denominator to allow for new categorical levels in the training set:\n",
    "\n",
    "$$\\phi_{j|y=1}= \\frac{\\Sigma_{i=1}^m 1(x_{j}^{(i)} \\ = \\ 1 \\ \\bigcap y^{(i)} \\ = \\ 1) \\ + \\ 1}{\\Sigma_{i=1}^{m}1(y^{(i)} \\ = \\ 1) \\ + \\ k}$$\n",
    "\n",
    "$$\\phi_{j|y=0}= \\frac{\\Sigma_{i=1}^m 1(x_{j}^{(i)} \\ = \\ 1 \\ \\bigcap y^{(i)} \\ = \\ 0) \\ + \\ 1}{\\Sigma_{i \\ = \\ 1}^{m}1(y^{(i)} \\ = \\ 0) \\ + \\ k}$$\n",
    "   \n",
    "(Where $x^{(i)}$ represents features, $y^{(i)}$ represents the response column, and $k$ represents the addition of each new categorical level (k functions to balance the added 1 in the numerator))\n",
    "\n",
    "Laplace smoothing should be used with care; it is generally intended to allow for predictions in rare events. As prediction data becomes increasingly distinct from training data, new models should be trained when possible to account for a broader set of possible feature values.\n",
    "\n",
    "  *After running the cells to load, clean, and split the data you can play with a Naive Bayes model [here](#NB Model).*\n",
    "\n",
    "<a name=\"PCA\"></a>\n",
    " ** PCA **: \n",
    "\n",
    "  New categorical levels in the test data that were not present in the training data, are skipped in the row product-  sum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"missing\"></a>\n",
    "## How Does H2O Handle Missing Values during Training & Testing? \n",
    "\n",
    "### Skip to the algorithm you're using to see how it trains or predicts with missing values:\n",
    "\n",
    "(Note: NA values in the training set are not neccessarily handled the same way as NA values in the test set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "* [Generalized Linear Model (GLM)](#mGLM)\n",
    "* [Gradient Boosting Model (GBM) & Distributed Random Forest (DRF)](#mGBM & DRF)\n",
    "* [Deep Learning](#mDL)\n",
    "* [K-Means](#mK)\n",
    "* [Naive Bayes](#mNB)\n",
    "* [Principal Component Analysis (PCA)](#mPCA)\n",
    "\n",
    "\n",
    "___\n",
    "\n",
    "<a name=\"mGLM\"></a>\n",
    " ** GLM **:\n",
    "\n",
    "  *How does the algorithm handle missing values during training?*\n",
    "  \n",
    "  Depending on the selected missing value handling policy, they are either imputed mean or the whole row is skipped.\n",
    "  The default behavior is mean imputation. Note that categorical variables are imputed by adding extra \"missing\"\n",
    "  level.\n",
    "  \n",
    "  Optionally, glm can skip all rows with any missing values.\n",
    "  \n",
    "  *How does the algorithm handle missing values during testing?* \n",
    "  \n",
    "  Same as during training. If the missing value handling is set to skip and we are generating predictions, skipped\n",
    "  rows will have Na (missing) prediction.\n",
    "\n",
    "<a name=\"mGBM & DRF\"></a>\n",
    " ** DRF & GBM ** :\n",
    "\n",
    "\n",
    "  *How does the algorithm handle missing values during training and testing?*\n",
    "  \n",
    "  Missing values always go right at every split decision.\n",
    " \n",
    "  \n",
    "<a name=\"mDL\"></a>\n",
    " ** Deep Learning **:\n",
    " \n",
    "  *How does the algorithm handle missing values during training?*\n",
    "  \n",
    "  Missing values in the training set will be mean-imputed or the whole row can be skipped, depending on how the\n",
    "  following parameter is set: <code>missing_values_handling = \"MeanImputation\" or \"Skip\"</code>.\n",
    "\n",
    "   *How does the algorithm handle missing values during testing?*\n",
    "   \n",
    "   Missing values in the test set will be mean-imputed (with the mean of the training data) during scoring.\n",
    "\n",
    "\n",
    "<a name=\"mK\"></a>\n",
    " ** K-Means **: \n",
    "\n",
    "  *How does the algorithm handle missing values during training?*\n",
    "  \n",
    "  Missing values are automatically imputed by the column mean. K-means also handles missing values by assuming that\n",
    "  missing feature distance contributions are equal to the average of all other distance term contributions.\n",
    "  \n",
    "  *How does the algorithm handle missing values during testing?*\n",
    "  \n",
    "  Missing values are automatically imputed by the column mean of the training data.\n",
    "\n",
    "<a name=\"mNB\"></a>\n",
    " ** Naive Bayes **:\n",
    "  \n",
    "  *How does the algorithm handle missing values during training?*\n",
    "  \n",
    "  All rows with one or more missing values (either in the predictors or the response) will be skipped during model building.\n",
    "  \n",
    "  *How does the algorithm handle missing values during testing?*\n",
    "  \n",
    "  If a predictor is missing, it will be skipped when taking the product of conditional probabilities in calculating\n",
    "  the joint probability conditional on the response.\n",
    "\n",
    "<a name=\"mPCA\"></a>\n",
    " ** PCA **: \n",
    " \n",
    "  *How does the algorithm handle missing values during scoring?*\n",
    "    \n",
    "  For the GramSVD and Power methods, all rows containing missing values are ignored during training. For the GLRM\n",
    "  method, missing values are excluded from the sum over the loss function in the objective. For more information,\n",
    "  refer to section 4 Generalized Loss Functions, equation (13), in \"Generalized Low Rank Models\" by Boyd et al.\n",
    "  \n",
    "  *How does the algorithm handle missing values during testing?*\n",
    "  \n",
    "  During scoring, the test data is right-multiplied by the eigenvector matrix produced by PCA. Missing categorical\n",
    "  values are skipped in the row product-sum. Missing numeric values propagate an entire row of NAs in the resulting\n",
    "  projection matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"resources\"></a>\n",
    "## Additional Resources\n",
    "0. [H2O's How To for Basic Questions](http://h2oai.github.io/data-science-examples/examples.html)\n",
    "1. [H2O's User Guide Documentation](http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/docs-website/h2o-docs/index.html)\n",
    "2. [H2O's Algorithm Documentation](https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/product/tutorials/datascience/DataScienceH2O-Dev.md)\n",
    "3. [H2O's Community - Where Questions are Posted](https://groups.google.com/forum/#!forum/h2ostream)\n",
    "4. [H2O's Python Demos](https://github.com/h2oai/h2o-3/tree/master/h2o-py/demos)\n",
    "5. [Python to R Parity](https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/product/upgrade/PythonParity.md)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
